Self-Driving Car Engineer Nanodegree

Deep Learning

Project: Build a Traffic Sign Recognition Classifier ---- Luke Xu

In this notebook, I build a traffic sign recognition classifier based on the German Traffic sign data set. I used several data preprocessing techniques, compared two different CNN structure and received close to human performance on the testing set.

In addition to implementing code, I did many experiments to get a better understanding of overfitting, underfitting and the effect of various hyper parameters.

At the end I get test accuracy is 97.22%


Step 0: Load The Data

In [1]:
# Load pickled data
import pickle

training_file = 'train.p'
testing_file = 'test.p'

with open(training_file, mode='rb') as f:
    train = pickle.load(f)
with open(testing_file, mode='rb') as f:
    test = pickle.load(f)
    
X_train, y_train = train['features'], train['labels']
X_test, y_test = test['features'], test['labels']

Step 1: Dataset Summary & Exploration

The pickled data is a dictionary with 4 key/value pairs:

  • 'features' is a 4D array containing raw pixel data of the traffic sign images, (num examples, width, height, channels).
  • 'labels' is a 2D array containing the label/class id of the traffic sign. The file signnames.csv contains id -> name mappings for each id.
  • 'sizes' is a list containing tuples, (width, height) representing the the original width and height the image.
  • 'coords' is a list containing tuples, (x1, y1, x2, y2) representing coordinates of a bounding box around the sign in the image. THESE COORDINATES ASSUME THE ORIGINAL IMAGE. THE PICKLED DATA CONTAINS RESIZED VERSIONS (32 by 32) OF THESE IMAGES

Complete the basic data summary below.

In [2]:
# Get validation data from training
from sklearn.model_selection import train_test_split
X_train, X_validation, y_train, y_validation = train_test_split(
    X_train, y_train, test_size=0.25, random_state=42)
assert(len(X_train) == len(y_train))
assert(len(X_validation) == len(y_validation))
assert(len(X_test) == len(y_test))

n_train = len(X_train)
n_test = len(X_test)
image_shape = X_train[0].shape
n_classes = len(set(list(y_train)))

print("Number of training examples =", n_train)
print("Number of testing examples =", n_test)
print("Image data shape =", image_shape)
print("Number of classes =", n_classes)
Number of training examples = 29406
Number of testing examples = 12630
Image data shape = (32, 32, 3)
Number of classes = 43

Visualize the German Traffic Signs Dataset using the pickled file(s). This is open ended, suggestions include: plotting traffic sign images, plotting the count of each sign, etc.

The Matplotlib examples and gallery pages are a great resource for doing visualizations in Python.

NOTE: It's recommended you start with something simple first. If you wish to do more, come back to it after you've completed the rest of the sections.

In [3]:
import random
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
#from augmentation import transform_image
import cv2

plt.hist(y_train,n_classes,alpha=0.5)
plt.hist(y_test,n_classes,alpha=0.5)
plt.title("Frequency of different traffic signs ")
plt.xlabel("Traffic signs")
plt.ylabel("Frequency")
plt.legend(('Training', 'Testing'))
plt.show()  

Step 2: Design and Test a Model Architecture

Design and implement a deep learning model that learns to recognize traffic signs. Train and test your model on the German Traffic Sign Dataset.

There are various aspects to consider when thinking about this problem:

  • Neural network architecture
  • Play around preprocessing techniques (normalization, rgb to grayscale, etc)
  • Number of examples per label (some have more than others).
  • Generate fake data.

Here is an example of a published baseline model on this problem. It's not required to be familiar with the approach used in the paper but, it's good practice to try to read papers like these.

NOTE: The LeNet-5 implementation shown in the classroom at the end of the CNN lesson is a solid starting point. You'll have to change the number of classes and possibly the preprocessing, but aside from that it's plug and play!

Implementation

Use the code cell (or multiple code cells, if necessary) to implement the first step of your project. Once you have completed your implementation and are satisfied with the results, be sure to thoroughly answer the questions that follow.

In [4]:
def CLAHE_transform(img):
    #Converting image to LAB Color model
    lab= cv2.cvtColor(img, cv2.COLOR_BGR2LAB)
    #Splitting the LAB image to different channels
    l, a, b = cv2.split(lab)
    #Applying CLAHE to L-channel
    clahe = cv2.createCLAHE(clipLimit=5, tileGridSize=(4,4))
    cl = clahe.apply(l)
    #Merge the CLAHE enhanced L-channel with the a and b channel
    limg = cv2.merge((cl,a,b))
    #Converting image from LAB Color model to RGB model
    final = cv2.cvtColor(limg, cv2.COLOR_LAB2BGR)
    return final

def plot_image(data = X_train,label = y_train,
               title = 'German Traffic Signs Dateset',
               NUM_OF_IMAGE =20,transform = False ):
    #Randomly find n images for each class
    
    #flag used to find n images for each class
    n=[0]*len(sign_list)
    image = [None]*len(sign_list)
    # Add image with the same index together as a row 
    # Until all index has at least N images
    while sum(n)<NUM_OF_IMAGE*len(sign_list):
        index = random.randint(0, len(data)-1)
        for j in range(len(sign_list)):
            if label[index] == j:
                if n[j] == 0:
                    image[j] = data[index]
                    if transform:
                        image[j]=CLAHE_transform(image[j])
                    else:
                        image[j]=image[j]
                elif n[j] < NUM_OF_IMAGE:
                    if transform:
                        image[j] = np.append(image[j],CLAHE_transform(data[index]),axis =1)
                    else:
                        image[j] = np.append(image[j],data[index],axis =1)
                else:
                    continue
                n[j]+=1
    img = image[0]
    for i in range(len(sign_list)-1):
        img = np.append(img,image[i+1],axis = 0)

    y_pos = np.arange(n_classes)*32+16

    plt.figure(figsize=(120,20))
    plt.yticks(y_pos, sign_list)
    plt.ylabel('Sign label abbreviations')
    plt.title(title)
    plt.imshow(img)
    
#Looking into the categories and use shorter names to represent the meaning.
import pandas as pd
names = pd.read_csv('signnames.csv')
sign_list=['20','30','50','60','70','80','End80','100','120','NoPa','NP>3.5',
     'RoW+','PrioR','Yield','Stop','NCar','N>3.5','NEnter','Caution','DgsCvL','DgsCvR',
     'DbCv','Bumpy','Slippery','NarowR','RoadWrk','Signal','Pdstrain','Child','Bike','ice',
     'animal','Edlim','TnR','TnL','AheadO','AorR','AorL','KeepR','KeepL','Round','ENPass','EPass>3.5']
print(len(names),len(sign_list))

#plot image before and after transform
plot_image()
plot_image(title = 'After CLAHE transform', transform = True)
43 43
In [5]:
def transform_image(img,ang_range=10,shear_range=5,trans_range=10):
    '''
    This function transforms images to generate new images.
    The function takes in following arguments,
    1- Image
    2- ang_range: Range of angles for rotation
    3- shear_range: Range of values to apply affine transform to
    4- trans_range: Range of values to apply translations over. 
    
    A Random uniform distribution is used to generate different parameters for transformation
    '''   
    # Rotation

    ang_rot = np.random.uniform(ang_range)-ang_range/2
    rows,cols,ch = img.shape    
    Rot_M = cv2.getRotationMatrix2D((cols/2,rows/2),ang_rot,1)

    # Translation
    tr_x = trans_range*np.random.uniform()-trans_range/2
    tr_y = trans_range*np.random.uniform()-trans_range/2
    Trans_M = np.float32([[1,0,tr_x],[0,1,tr_y]])

    # Shear
    pts1 = np.float32([[5,5],[20,5],[5,20]])

    pt1 = 5+shear_range*np.random.uniform()-shear_range/2
    pt2 = 20+shear_range*np.random.uniform()-shear_range/2
    
    # Brightness 
    

    pts2 = np.float32([[pt1,5],[pt2,pt1],[5,pt2]])

    shear_M = cv2.getAffineTransform(pts1,pts2)
        
    img = cv2.warpAffine(img,Rot_M,(cols,rows))
    img = cv2.warpAffine(img,Trans_M,(cols,rows))
    img = cv2.warpAffine(img,shear_M,(cols,rows))
    
    return img


augmented_X = []
augmented_y = []


    # Apply CLAHE transform to X_test, X_validation as well
for index in range(len(X_test)):
    X_test[index] = CLAHE_transform(X_test[index])
for index in range(len(X_validation)):
    X_validation[index] =CLAHE_transform(X_validation[index])

for index in range(len(X_train)):
    #Add original image and index
    augmented_X.append(CLAHE_transform(X_train[index]))
    augmented_y.append(y_train[index])

    for i in range(8):
        #Add more data 
        augmented_X.append(transform_image(CLAHE_transform(X_train[index])))
        augmented_y.append(y_train[index])
In [6]:
plot_image(data = augmented_X,label = augmented_y,title='Augmented dataset')
print(len(augmented_X))
print(len(X_train))
264654
29406
In [7]:
from sklearn.utils import shuffle
X_train, y_train = shuffle(augmented_X, augmented_y)

#X_train, y_train = shuffle(X_train, y_train)
print(len(X_train),len(y_train))

import tensorflow as tf
from tensorflow.contrib.layers import flatten
plt.imshow(X_train[-1])
sign_list[int(y_train[-1])]
264654 264654
Out[7]:
'DgsCvR'
In [36]:
def LeNet(x,dropout):    
    # Hyperparameters
    mu = 0
    sigma = 0.1
    
    # SOLUTION: Layer 1: Convolutional. Input = 32x32x1. Output = 28x28x6.
    conv1_W = tf.Variable(tf.truncated_normal(shape=(5, 5, 3, 6), mean = mu, stddev = sigma))
    conv1_b = tf.Variable(tf.zeros(6))
    conv1   = tf.nn.conv2d(x, conv1_W, strides=[1, 1, 1, 1], padding='VALID') + conv1_b
    # SOLUTION: Activation.
    conv1 = tf.nn.relu(conv1)
    # SOLUTION: Pooling. Input = 28x28x6. Output = 14x14x6.
    conv1 = tf.nn.max_pool(conv1, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='VALID')

    
    
    # SOLUTION: Layer 2: Convolutional. Output = 10x10x16.
    conv2_W = tf.Variable(tf.truncated_normal(shape=(5, 5, 6, 16), mean = mu, stddev = sigma))
    conv2_b = tf.Variable(tf.zeros(16))
    conv2   = tf.nn.conv2d(conv1, conv2_W, strides=[1, 1, 1, 1], padding='VALID') + conv2_b
    # SOLUTION: Activation.
    conv2 = tf.nn.relu(conv2)
    # SOLUTION: Pooling. Input = 10x10x16. Output = 5x5x16.
    conv2 = tf.nn.max_pool(conv2, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='VALID')

    # SOLUTION: Flatten. Input = 5x5x16. Output = 400.
    fc0   = flatten(conv2)
    
    # SOLUTION: Layer 3: Fully Connected. Input = 400. Output = 120.
    fc1_W = tf.Variable(tf.truncated_normal(shape=(400, 120), mean = mu, stddev = sigma))
    fc1_b = tf.Variable(tf.zeros(120))
    fc1   = tf.matmul(fc0, fc1_W) + fc1_b
    
    # SOLUTION: Activation.
    fc1    = tf.nn.relu(fc1)
    fc1 = tf.nn.dropout(fc1, dropout)

    # SOLUTION: Layer 4: Fully Connected. Input = 120. Output = 84.
    fc2_W  = tf.Variable(tf.truncated_normal(shape=(120, 84), mean = mu, stddev = sigma))
    fc2_b  = tf.Variable(tf.zeros(84))
    fc2    = tf.matmul(fc1, fc2_W) + fc2_b
    
    # SOLUTION: Activation.
    fc2    = tf.nn.relu(fc2)

    # SOLUTION: Layer 5: Fully Connected. Input = 84. Output = 43.
    fc3_W  = tf.Variable(tf.truncated_normal(shape=(84, 43), mean = mu, stddev = sigma))
    fc3_b  = tf.Variable(tf.zeros(43))
    logits = tf.matmul(fc2, fc3_W) + fc3_b
    
    return logits
In [8]:
from tensorflow.contrib.layers import flatten

def conv_net(x,dropout = 0):    
    # Hyperparameters
    mu = 0
    sigma = 0.1
    
    # SOLUTION: Layer 1: Convolutional. Input = 32x32x1. Output = 28x28x100.
    conv1_W = tf.Variable(tf.truncated_normal(shape=(5, 5, 3, 100), mean = mu, stddev = sigma))
    conv1_b = tf.Variable(tf.zeros(100))
    conv1   = tf.nn.conv2d(x, conv1_W, strides=[1, 1, 1, 1], padding='VALID') + conv1_b
    conv1 = tf.nn.relu(conv1)
    # SOLUTION: Pooling. Input = 28x28x100. Output = 14x14x100.
    conv1 = tf.nn.max_pool(conv1, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='VALID')

    
    # SOLUTION: Layer 2: Convolutional. Output = 12x12x150.
    conv2_W = tf.Variable(tf.truncated_normal(shape=(3, 3, 100, 150), mean = mu, stddev = sigma))
    conv2_b = tf.Variable(tf.zeros(150))
    conv2   = tf.nn.conv2d(conv1, conv2_W, strides=[1, 1, 1, 1], padding='VALID') + conv2_b
    # SOLUTION: Activation.
    conv2 = tf.nn.relu(conv2)
    # SOLUTION: Pooling. Input =12x12x150.. Output = 6x6x150.
    conv2 = tf.nn.max_pool(conv2, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='VALID')
    
    # SOLUTION: Layer 3: Convolutional. Output = 5x5x250.
    conv3_W = tf.Variable(tf.truncated_normal(shape=(2, 2, 150, 250), mean = mu, stddev = sigma))
    conv3_b = tf.Variable(tf.zeros(250))
    conv3   = tf.nn.conv2d(conv2, conv3_W, strides=[1, 1, 1, 1], padding='VALID') + conv3_b
    # SOLUTION: Activation.
    conv3 = tf.nn.relu(conv3)
    # SOLUTION: Pooling. Input = 5x5x250. Output =2x2x250.
    conv3 = tf.nn.max_pool(conv3, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='VALID')
    
    
    # SOLUTION: Flatten. Input = 2x2x250. Output = 300.
    fc0   = flatten(conv3)
       
    
    
    # SOLUTION: Layer 3: Fully Connected. Input = 1000. Output = 300.
    fc1_W = tf.Variable(tf.truncated_normal(shape=(1000, 300), mean = mu, stddev = sigma))
    fc1_b = tf.Variable(tf.zeros(300))
    fc1   = tf.matmul(fc0, fc1_W) + fc1_b
    # SOLUTION: Activation.
    fc1    = tf.nn.relu(fc1)
    fc1 = tf.nn.dropout(fc1, dropout)

    # SOLUTION: Layer 4: Fully Connected. Input = 300. Output = 100.
    fc2_W  = tf.Variable(tf.truncated_normal(shape=(300, 43), mean = mu, stddev = sigma))
    fc2_b  = tf.Variable(tf.zeros(43))
    logits    = tf.matmul(fc1, fc2_W) + fc2_b
    #logits = fc2
    
#     # SOLUTION: Activation.
#     fc2    = tf.nn.relu(fc2)

#     # SOLUTION: Layer 5: Fully Connected. Input = 100. Output = 43.
#     fc3_W  = tf.Variable(tf.truncated_normal(shape=(100, 43), mean = mu, stddev = sigma))
#     fc3_b  = tf.Variable(tf.zeros(43))
#     logits = tf.matmul(fc2, fc3_W) + fc3_b
    
    return logits
In [9]:
rand = 10**(-np.random.uniform(3,6,10))
In [ ]:
rand
Out[ ]:
array([  1.00009853e-06,   2.53323187e-06,   2.02059829e-05,
         4.17324987e-04,   5.66564425e-05,   9.74566992e-04,
         5.16106526e-04,   2.97530451e-05,   1.02637939e-04,
         1.19830024e-04])
In [ ]:
# Random search for the learning rate
data = []
best_validation_accuracy = 0.9
for i in rand:
    data_list = [i]
    x = tf.placeholder(tf.float32, (None, 32, 32, 3))
    y = tf.placeholder(tf.int32, (None))
    one_hot_y = tf.one_hot(y, 43)
    keep_prob = tf.placeholder(tf.float32)

    rate = i
    print('learning rate = ', rate)
    logits = conv_net(x,keep_prob)
    #logits = LeNet(x,keep_prob)
    cross_entropy = tf.nn.softmax_cross_entropy_with_logits(logits, one_hot_y)
    loss_operation = tf.reduce_mean(cross_entropy)
    optimizer = tf.train.AdamOptimizer(learning_rate = rate)
    training_operation = optimizer.minimize(loss_operation)


    correct_prediction = tf.equal(tf.argmax(logits, 1), tf.argmax(one_hot_y, 1))
    accuracy_operation = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
    saver = tf.train.Saver()

    #Add loss in the evaluate function
    def evaluate(X_data, y_data):
        num_examples = len(X_data)
        total_accuracy = 0
        total_loss = 0
        sess = tf.get_default_session()
        for offset in range(0, num_examples, BATCH_SIZE):
            batch_x, batch_y = X_data[offset:offset+BATCH_SIZE], y_data[offset:offset+BATCH_SIZE]
            accuracy,_loss = sess.run([accuracy_operation,loss_operation], 
                                      feed_dict={x: batch_x, y: batch_y,keep_prob: 1.0})
            total_accuracy += (accuracy * len(batch_x))
            total_loss += (_loss*len(batch_x))
        return(total_accuracy / num_examples, total_loss / num_examples)

    EPOCHS = 50
    BATCH_SIZE = 128
    DROPOUT = 0.8
    EARLY_STOP = 5


    with tf.Session() as sess:
        sess.run(tf.global_variables_initializer())
        num_examples = len(X_train)
        print("Training...")
        print()
        train_acc = []
        valid_acc = []
        train_loss = []
        valid_loss = []
        for i in range(EPOCHS):
            X_train, y_train = shuffle(X_train, y_train)
            for offset in range(0, num_examples, BATCH_SIZE):
                end = offset + BATCH_SIZE
                batch_x, batch_y = X_train[offset:end], y_train[offset:end]
                sess.run(training_operation, feed_dict={x: batch_x, y: batch_y,keep_prob:DROPOUT})

            training_accuracy,training_loss   =   evaluate(X_train, y_train)
            validation_accuracy,validation_loss = evaluate(X_validation, y_validation)
            train_acc.append(training_accuracy)
            valid_acc.append(validation_accuracy)
            train_loss.append(training_loss)
            valid_loss.append(validation_loss)
            print("EPOCH {} ...".format(i+1))
            print("Train Accuracy = {:.4f} ~".format(training_accuracy),
                  "Validation Accuracy = {:.3f}".format(validation_accuracy),
                  "Train loss = {:.4f}".format(training_loss),
                  "Validation loss = {:.3f}".format(validation_loss))
            if len(valid_loss)>6 and \
            valid_loss[-1]>= valid_loss[-2] and valid_loss[-1] >= np.mean(valid_loss[-EARLY_STOP-1:]) and\
            valid_loss[-2]>= valid_loss[-3] and valid_loss[-2] >= np.mean(valid_loss[-EARLY_STOP-2:-2]) :
                print('Early stop, last five validation loss mean is ',np.mean(valid_loss[-EARLY_STOP:]))
                break
        data_list.append(train_loss)
        data_list.append(valid_loss)
        plt.plot(train_loss, 'b--',valid_loss,'ro')
        plt.show()
        plt.plot(train_acc, 'b--',valid_acc,'ro')
        plt.show()
        if valid_loss[-1] > best_validation_accuracy:
            saver.save(sess, 'best_net')
            print("Model saved")
    data.append(data_list)
learning rate =  1.00009852798e-06
Training...

EPOCH 1 ...
Train Accuracy = 0.0560 ~ Validation Accuracy = 0.064 Train loss = 295.7520 Validation loss = 301.431
EPOCH 2 ...
Train Accuracy = 0.0707 ~ Validation Accuracy = 0.097 Train loss = 165.3482 Validation loss = 168.575
EPOCH 3 ...
Train Accuracy = 0.0777 ~ Validation Accuracy = 0.108 Train loss = 103.1480 Validation loss = 105.236
EPOCH 4 ...
Train Accuracy = 0.0794 ~ Validation Accuracy = 0.108 Train loss = 55.2990 Validation loss = 57.384
EPOCH 5 ...
Train Accuracy = 0.0476 ~ Validation Accuracy = 0.067 Train loss = 15.6640 Validation loss = 16.807
EPOCH 6 ...
Train Accuracy = 0.0550 ~ Validation Accuracy = 0.063 Train loss = 6.1534 Validation loss = 6.165
EPOCH 7 ...
Train Accuracy = 0.0537 ~ Validation Accuracy = 0.063 Train loss = 4.5699 Validation loss = 4.434
EPOCH 8 ...
Train Accuracy = 0.0520 ~ Validation Accuracy = 0.060 Train loss = 4.1096 Validation loss = 4.002
EPOCH 9 ...
Train Accuracy = 0.0512 ~ Validation Accuracy = 0.060 Train loss = 3.9294 Validation loss = 3.868
EPOCH 10 ...
Train Accuracy = 0.0512 ~ Validation Accuracy = 0.060 Train loss = 3.8499 Validation loss = 3.812
EPOCH 11 ...
Train Accuracy = 0.0514 ~ Validation Accuracy = 0.060 Train loss = 3.8092 Validation loss = 3.779
EPOCH 12 ...
Train Accuracy = 0.0522 ~ Validation Accuracy = 0.061 Train loss = 3.7857 Validation loss = 3.761
EPOCH 13 ...
Train Accuracy = 0.0526 ~ Validation Accuracy = 0.062 Train loss = 3.7705 Validation loss = 3.745
EPOCH 14 ...
Train Accuracy = 0.0535 ~ Validation Accuracy = 0.063 Train loss = 3.7599 Validation loss = 3.735
EPOCH 15 ...
Train Accuracy = 0.0539 ~ Validation Accuracy = 0.064 Train loss = 3.7522 Validation loss = 3.727
EPOCH 16 ...
Train Accuracy = 0.0539 ~ Validation Accuracy = 0.064 Train loss = 3.7468 Validation loss = 3.723
EPOCH 17 ...
Train Accuracy = 0.0539 ~ Validation Accuracy = 0.064 Train loss = 3.7415 Validation loss = 3.715
EPOCH 18 ...
Train Accuracy = 0.0546 ~ Validation Accuracy = 0.066 Train loss = 3.7369 Validation loss = 3.707
EPOCH 19 ...
Train Accuracy = 0.0548 ~ Validation Accuracy = 0.066 Train loss = 3.7325 Validation loss = 3.702
EPOCH 20 ...
Train Accuracy = 0.0557 ~ Validation Accuracy = 0.068 Train loss = 3.7275 Validation loss = 3.693
EPOCH 21 ...
Train Accuracy = 0.0563 ~ Validation Accuracy = 0.069 Train loss = 3.7231 Validation loss = 3.686
EPOCH 22 ...
Train Accuracy = 0.0571 ~ Validation Accuracy = 0.071 Train loss = 3.7179 Validation loss = 3.676
EPOCH 23 ...
Train Accuracy = 0.0582 ~ Validation Accuracy = 0.073 Train loss = 3.7124 Validation loss = 3.665
EPOCH 24 ...
Train Accuracy = 0.0607 ~ Validation Accuracy = 0.078 Train loss = 3.7058 Validation loss = 3.650
EPOCH 25 ...
Train Accuracy = 0.0622 ~ Validation Accuracy = 0.082 Train loss = 3.6998 Validation loss = 3.639
EPOCH 26 ...
Train Accuracy = 0.0633 ~ Validation Accuracy = 0.084 Train loss = 3.6931 Validation loss = 3.627
EPOCH 27 ...
Train Accuracy = 0.0636 ~ Validation Accuracy = 0.084 Train loss = 3.6881 Validation loss = 3.615
EPOCH 28 ...
Train Accuracy = 0.0660 ~ Validation Accuracy = 0.087 Train loss = 3.6798 Validation loss = 3.602
EPOCH 29 ...
Train Accuracy = 0.0699 ~ Validation Accuracy = 0.095 Train loss = 3.6690 Validation loss = 3.584
EPOCH 30 ...
Train Accuracy = 0.0714 ~ Validation Accuracy = 0.096 Train loss = 3.6616 Validation loss = 3.573
EPOCH 31 ...
Train Accuracy = 0.0749 ~ Validation Accuracy = 0.098 Train loss = 3.6517 Validation loss = 3.560
EPOCH 32 ...
Train Accuracy = 0.0854 ~ Validation Accuracy = 0.118 Train loss = 3.6415 Validation loss = 3.547
EPOCH 33 ...
Train Accuracy = 0.0884 ~ Validation Accuracy = 0.121 Train loss = 3.6318 Validation loss = 3.536
EPOCH 34 ...
Train Accuracy = 0.0887 ~ Validation Accuracy = 0.121 Train loss = 3.6248 Validation loss = 3.525
EPOCH 35 ...
Train Accuracy = 0.0947 ~ Validation Accuracy = 0.128 Train loss = 3.6123 Validation loss = 3.508
EPOCH 36 ...
Train Accuracy = 0.0918 ~ Validation Accuracy = 0.125 Train loss = 3.6093 Validation loss = 3.504
EPOCH 37 ...
Train Accuracy = 0.0963 ~ Validation Accuracy = 0.130 Train loss = 3.5972 Validation loss = 3.491
EPOCH 38 ...
Train Accuracy = 0.0988 ~ Validation Accuracy = 0.134 Train loss = 3.5895 Validation loss = 3.483
EPOCH 39 ...
Train Accuracy = 0.0985 ~ Validation Accuracy = 0.133 Train loss = 3.5862 Validation loss = 3.475
EPOCH 40 ...
Train Accuracy = 0.1093 ~ Validation Accuracy = 0.138 Train loss = 3.5761 Validation loss = 3.469
EPOCH 41 ...
Train Accuracy = 0.1103 ~ Validation Accuracy = 0.139 Train loss = 3.5697 Validation loss = 3.458
EPOCH 42 ...
Train Accuracy = 0.1138 ~ Validation Accuracy = 0.143 Train loss = 3.5607 Validation loss = 3.448
EPOCH 43 ...
Train Accuracy = 0.1139 ~ Validation Accuracy = 0.143 Train loss = 3.5564 Validation loss = 3.443
EPOCH 44 ...
Train Accuracy = 0.1163 ~ Validation Accuracy = 0.146 Train loss = 3.5486 Validation loss = 3.437
EPOCH 45 ...
Train Accuracy = 0.1192 ~ Validation Accuracy = 0.149 Train loss = 3.5400 Validation loss = 3.425
EPOCH 46 ...
Train Accuracy = 0.1215 ~ Validation Accuracy = 0.151 Train loss = 3.5327 Validation loss = 3.420
EPOCH 47 ...
Train Accuracy = 0.1179 ~ Validation Accuracy = 0.147 Train loss = 3.5330 Validation loss = 3.420
EPOCH 48 ...
Train Accuracy = 0.1213 ~ Validation Accuracy = 0.150 Train loss = 3.5232 Validation loss = 3.410
EPOCH 49 ...
Train Accuracy = 0.1217 ~ Validation Accuracy = 0.151 Train loss = 3.5193 Validation loss = 3.404
EPOCH 50 ...
Train Accuracy = 0.1255 ~ Validation Accuracy = 0.154 Train loss = 3.5097 Validation loss = 3.397
Model saved
learning rate =  2.53323186503e-06
Training...

EPOCH 1 ...
Train Accuracy = 0.0849 ~ Validation Accuracy = 0.137 Train loss = 150.8821 Validation loss = 161.148
EPOCH 2 ...
Train Accuracy = 0.0527 ~ Validation Accuracy = 0.064 Train loss = 27.2654 Validation loss = 30.167
EPOCH 3 ...
Train Accuracy = 0.0415 ~ Validation Accuracy = 0.040 Train loss = 5.5769 Validation loss = 5.508
EPOCH 4 ...
Train Accuracy = 0.0497 ~ Validation Accuracy = 0.048 Train loss = 4.2359 Validation loss = 4.161
EPOCH 5 ...
Train Accuracy = 0.0523 ~ Validation Accuracy = 0.052 Train loss = 3.9170 Validation loss = 3.886
EPOCH 6 ...
Train Accuracy = 0.0533 ~ Validation Accuracy = 0.053 Train loss = 3.8101 Validation loss = 3.803
EPOCH 7 ...
Train Accuracy = 0.0586 ~ Validation Accuracy = 0.058 Train loss = 3.7714 Validation loss = 3.773
EPOCH 8 ...
Train Accuracy = 0.0610 ~ Validation Accuracy = 0.060 Train loss = 3.7507 Validation loss = 3.755
EPOCH 9 ...
Train Accuracy = 0.0642 ~ Validation Accuracy = 0.067 Train loss = 3.7331 Validation loss = 3.733
EPOCH 10 ...
Train Accuracy = 0.0683 ~ Validation Accuracy = 0.074 Train loss = 3.7140 Validation loss = 3.697
EPOCH 11 ...
Train Accuracy = 0.0713 ~ Validation Accuracy = 0.079 Train loss = 3.6980 Validation loss = 3.675
EPOCH 12 ...
Train Accuracy = 0.0752 ~ Validation Accuracy = 0.085 Train loss = 3.6816 Validation loss = 3.652
EPOCH 13 ...
Train Accuracy = 0.0783 ~ Validation Accuracy = 0.091 Train loss = 3.6675 Validation loss = 3.630
EPOCH 14 ...
Train Accuracy = 0.0810 ~ Validation Accuracy = 0.094 Train loss = 3.6554 Validation loss = 3.610
EPOCH 15 ...
Train Accuracy = 0.0826 ~ Validation Accuracy = 0.096 Train loss = 3.6452 Validation loss = 3.598
EPOCH 16 ...
Train Accuracy = 0.0848 ~ Validation Accuracy = 0.098 Train loss = 3.6336 Validation loss = 3.586
EPOCH 17 ...
Train Accuracy = 0.0911 ~ Validation Accuracy = 0.108 Train loss = 3.6191 Validation loss = 3.561
EPOCH 18 ...
Train Accuracy = 0.1030 ~ Validation Accuracy = 0.121 Train loss = 3.5785 Validation loss = 3.510
EPOCH 19 ...
Train Accuracy = 0.1065 ~ Validation Accuracy = 0.123 Train loss = 3.5594 Validation loss = 3.492
EPOCH 20 ...
Train Accuracy = 0.1123 ~ Validation Accuracy = 0.127 Train loss = 3.5388 Validation loss = 3.473
EPOCH 21 ...
Train Accuracy = 0.1120 ~ Validation Accuracy = 0.127 Train loss = 3.5234 Validation loss = 3.462
EPOCH 22 ...
Train Accuracy = 0.1136 ~ Validation Accuracy = 0.128 Train loss = 3.5089 Validation loss = 3.452
EPOCH 23 ...
Train Accuracy = 0.1146 ~ Validation Accuracy = 0.128 Train loss = 3.4981 Validation loss = 3.444
EPOCH 24 ...
Train Accuracy = 0.1189 ~ Validation Accuracy = 0.130 Train loss = 3.4820 Validation loss = 3.423
EPOCH 25 ...
Train Accuracy = 0.1166 ~ Validation Accuracy = 0.130 Train loss = 3.4778 Validation loss = 3.427
EPOCH 26 ...
Train Accuracy = 0.1186 ~ Validation Accuracy = 0.130 Train loss = 3.4655 Validation loss = 3.416
EPOCH 27 ...
Train Accuracy = 0.1190 ~ Validation Accuracy = 0.130 Train loss = 3.4578 Validation loss = 3.407
EPOCH 28 ...
Train Accuracy = 0.1195 ~ Validation Accuracy = 0.130 Train loss = 3.4517 Validation loss = 3.404
EPOCH 29 ...
Train Accuracy = 0.1205 ~ Validation Accuracy = 0.130 Train loss = 3.4443 Validation loss = 3.399
EPOCH 30 ...
Train Accuracy = 0.1212 ~ Validation Accuracy = 0.131 Train loss = 3.4378 Validation loss = 3.394
EPOCH 31 ...
Train Accuracy = 0.1213 ~ Validation Accuracy = 0.130 Train loss = 3.4333 Validation loss = 3.390
EPOCH 32 ...
Train Accuracy = 0.1228 ~ Validation Accuracy = 0.131 Train loss = 3.4265 Validation loss = 3.389
EPOCH 33 ...
Train Accuracy = 0.1243 ~ Validation Accuracy = 0.131 Train loss = 3.4186 Validation loss = 3.380
EPOCH 34 ...
Train Accuracy = 0.1240 ~ Validation Accuracy = 0.131 Train loss = 3.4197 Validation loss = 3.388
EPOCH 35 ...
Train Accuracy = 0.1238 ~ Validation Accuracy = 0.131 Train loss = 3.4107 Validation loss = 3.376
EPOCH 36 ...
Train Accuracy = 0.1254 ~ Validation Accuracy = 0.132 Train loss = 3.4029 Validation loss = 3.370
EPOCH 37 ...
Train Accuracy = 0.1264 ~ Validation Accuracy = 0.132 Train loss = 3.4002 Validation loss = 3.370
EPOCH 38 ...
Train Accuracy = 0.1241 ~ Validation Accuracy = 0.131 Train loss = 3.4007 Validation loss = 3.368
EPOCH 39 ...
Train Accuracy = 0.1268 ~ Validation Accuracy = 0.132 Train loss = 3.3917 Validation loss = 3.365
EPOCH 40 ...
Train Accuracy = 0.1264 ~ Validation Accuracy = 0.132 Train loss = 3.3849 Validation loss = 3.358
EPOCH 41 ...
Train Accuracy = 0.1264 ~ Validation Accuracy = 0.132 Train loss = 3.3828 Validation loss = 3.358
EPOCH 42 ...
Train Accuracy = 0.1275 ~ Validation Accuracy = 0.132 Train loss = 3.3757 Validation loss = 3.353
EPOCH 43 ...
Train Accuracy = 0.1273 ~ Validation Accuracy = 0.132 Train loss = 3.3733 Validation loss = 3.350
EPOCH 44 ...
Train Accuracy = 0.1286 ~ Validation Accuracy = 0.133 Train loss = 3.3665 Validation loss = 3.345
EPOCH 45 ...
Train Accuracy = 0.1286 ~ Validation Accuracy = 0.132 Train loss = 3.3669 Validation loss = 3.351
EPOCH 46 ...
Train Accuracy = 0.1289 ~ Validation Accuracy = 0.132 Train loss = 3.3587 Validation loss = 3.342
EPOCH 47 ...
Train Accuracy = 0.1291 ~ Validation Accuracy = 0.132 Train loss = 3.3558 Validation loss = 3.339
EPOCH 48 ...
Train Accuracy = 0.1295 ~ Validation Accuracy = 0.133 Train loss = 3.3500 Validation loss = 3.335
EPOCH 49 ...
Train Accuracy = 0.1300 ~ Validation Accuracy = 0.132 Train loss = 3.3497 Validation loss = 3.337
EPOCH 50 ...
Train Accuracy = 0.1311 ~ Validation Accuracy = 0.133 Train loss = 3.3405 Validation loss = 3.329
Model saved
learning rate =  2.02059829053e-05
Training...

EPOCH 1 ...
Train Accuracy = 0.0573 ~ Validation Accuracy = 0.062 Train loss = 3.9752 Validation loss = 3.900
EPOCH 2 ...
Train Accuracy = 0.0644 ~ Validation Accuracy = 0.078 Train loss = 3.7429 Validation loss = 3.719
EPOCH 3 ...
Train Accuracy = 0.0740 ~ Validation Accuracy = 0.090 Train loss = 3.6698 Validation loss = 3.622
EPOCH 4 ...
Train Accuracy = 0.0829 ~ Validation Accuracy = 0.105 Train loss = 3.6161 Validation loss = 3.541
EPOCH 5 ...
Train Accuracy = 0.0968 ~ Validation Accuracy = 0.112 Train loss = 3.5639 Validation loss = 3.492
EPOCH 6 ...
Train Accuracy = 0.0959 ~ Validation Accuracy = 0.115 Train loss = 3.5298 Validation loss = 3.461
EPOCH 7 ...
Train Accuracy = 0.1209 ~ Validation Accuracy = 0.145 Train loss = 3.4377 Validation loss = 3.340
EPOCH 8 ...
Train Accuracy = 0.1425 ~ Validation Accuracy = 0.180 Train loss = 3.3250 Validation loss = 3.171
EPOCH 9 ...
Train Accuracy = 0.1629 ~ Validation Accuracy = 0.187 Train loss = 3.2321 Validation loss = 3.115
EPOCH 10 ...
Train Accuracy = 0.1761 ~ Validation Accuracy = 0.194 Train loss = 3.1768 Validation loss = 3.032
EPOCH 11 ...
Train Accuracy = 0.1790 ~ Validation Accuracy = 0.191 Train loss = 3.1188 Validation loss = 2.995
EPOCH 12 ...
Train Accuracy = 0.1815 ~ Validation Accuracy = 0.191 Train loss = 3.0790 Validation loss = 2.971
EPOCH 13 ...
Train Accuracy = 0.1855 ~ Validation Accuracy = 0.191 Train loss = 3.0495 Validation loss = 2.955
EPOCH 14 ...
Train Accuracy = 0.1897 ~ Validation Accuracy = 0.197 Train loss = 3.0237 Validation loss = 2.935
EPOCH 15 ...
Train Accuracy = 0.1989 ~ Validation Accuracy = 0.199 Train loss = 2.9772 Validation loss = 2.912
EPOCH 16 ...
Train Accuracy = 0.2021 ~ Validation Accuracy = 0.204 Train loss = 2.9687 Validation loss = 2.897
EPOCH 17 ...
Train Accuracy = 0.2050 ~ Validation Accuracy = 0.216 Train loss = 2.9593 Validation loss = 2.885
EPOCH 18 ...
Train Accuracy = 0.2009 ~ Validation Accuracy = 0.201 Train loss = 2.9309 Validation loss = 2.847
EPOCH 19 ...
Train Accuracy = 0.2114 ~ Validation Accuracy = 0.208 Train loss = 2.8938 Validation loss = 2.824
EPOCH 20 ...
Train Accuracy = 0.2179 ~ Validation Accuracy = 0.228 Train loss = 2.8928 Validation loss = 2.816
EPOCH 21 ...
Train Accuracy = 0.2319 ~ Validation Accuracy = 0.242 Train loss = 2.8474 Validation loss = 2.772
EPOCH 22 ...
Train Accuracy = 0.2280 ~ Validation Accuracy = 0.260 Train loss = 2.8542 Validation loss = 2.762
EPOCH 23 ...
Train Accuracy = 0.2467 ~ Validation Accuracy = 0.276 Train loss = 2.8285 Validation loss = 2.723
EPOCH 24 ...
Train Accuracy = 0.2489 ~ Validation Accuracy = 0.279 Train loss = 2.7960 Validation loss = 2.707
EPOCH 25 ...
Train Accuracy = 0.2649 ~ Validation Accuracy = 0.285 Train loss = 2.7516 Validation loss = 2.680
EPOCH 26 ...
Train Accuracy = 0.2735 ~ Validation Accuracy = 0.288 Train loss = 2.7261 Validation loss = 2.665
EPOCH 27 ...
Train Accuracy = 0.2752 ~ Validation Accuracy = 0.290 Train loss = 2.7153 Validation loss = 2.650
EPOCH 28 ...
Train Accuracy = 0.2790 ~ Validation Accuracy = 0.291 Train loss = 2.6982 Validation loss = 2.643
EPOCH 29 ...
Train Accuracy = 0.2850 ~ Validation Accuracy = 0.294 Train loss = 2.6753 Validation loss = 2.628
EPOCH 30 ...
Train Accuracy = 0.2829 ~ Validation Accuracy = 0.292 Train loss = 2.6740 Validation loss = 2.625
EPOCH 31 ...
Train Accuracy = 0.2896 ~ Validation Accuracy = 0.303 Train loss = 2.6631 Validation loss = 2.605
EPOCH 32 ...
Train Accuracy = 0.3017 ~ Validation Accuracy = 0.316 Train loss = 2.5985 Validation loss = 2.535
EPOCH 33 ...
Train Accuracy = 0.3010 ~ Validation Accuracy = 0.319 Train loss = 2.6127 Validation loss = 2.526
EPOCH 34 ...
Train Accuracy = 0.3116 ~ Validation Accuracy = 0.322 Train loss = 2.5413 Validation loss = 2.472
EPOCH 35 ...
Train Accuracy = 0.3090 ~ Validation Accuracy = 0.318 Train loss = 2.5117 Validation loss = 2.441
EPOCH 36 ...
Train Accuracy = 0.3098 ~ Validation Accuracy = 0.311 Train loss = 2.4737 Validation loss = 2.388
EPOCH 37 ...
Train Accuracy = 0.3278 ~ Validation Accuracy = 0.348 Train loss = 2.4598 Validation loss = 2.350
EPOCH 38 ...
Train Accuracy = 0.3291 ~ Validation Accuracy = 0.342 Train loss = 2.3981 Validation loss = 2.304
EPOCH 39 ...
Train Accuracy = 0.3266 ~ Validation Accuracy = 0.332 Train loss = 2.3707 Validation loss = 2.278
EPOCH 40 ...
Train Accuracy = 0.3449 ~ Validation Accuracy = 0.363 Train loss = 2.3271 Validation loss = 2.236
EPOCH 41 ...
Train Accuracy = 0.3596 ~ Validation Accuracy = 0.389 Train loss = 2.2723 Validation loss = 2.182
EPOCH 42 ...
Train Accuracy = 0.3632 ~ Validation Accuracy = 0.379 Train loss = 2.1806 Validation loss = 2.096
EPOCH 43 ...
Train Accuracy = 0.3655 ~ Validation Accuracy = 0.390 Train loss = 2.1530 Validation loss = 2.061
EPOCH 44 ...
Train Accuracy = 0.3693 ~ Validation Accuracy = 0.385 Train loss = 2.0956 Validation loss = 2.016
EPOCH 45 ...
Train Accuracy = 0.3756 ~ Validation Accuracy = 0.402 Train loss = 2.0780 Validation loss = 1.985
EPOCH 46 ...
Train Accuracy = 0.3787 ~ Validation Accuracy = 0.407 Train loss = 2.0223 Validation loss = 1.937
EPOCH 47 ...
Train Accuracy = 0.3904 ~ Validation Accuracy = 0.424 Train loss = 2.0003 Validation loss = 1.906
EPOCH 48 ...
Train Accuracy = 0.3935 ~ Validation Accuracy = 0.419 Train loss = 1.9311 Validation loss = 1.841
EPOCH 49 ...
Train Accuracy = 0.4080 ~ Validation Accuracy = 0.443 Train loss = 1.8906 Validation loss = 1.772
EPOCH 50 ...
Train Accuracy = 0.4111 ~ Validation Accuracy = 0.424 Train loss = 1.8411 Validation loss = 1.737
Model saved
learning rate =  0.000417324987047
Training...

EPOCH 1 ...
Train Accuracy = 0.0667 ~ Validation Accuracy = 0.078 Train loss = 3.5021 Validation loss = 3.482
EPOCH 2 ...
Train Accuracy = 0.0701 ~ Validation Accuracy = 0.080 Train loss = 3.4531 Validation loss = 3.416
EPOCH 3 ...
Train Accuracy = 0.1138 ~ Validation Accuracy = 0.138 Train loss = 3.3067 Validation loss = 3.212
EPOCH 4 ...
Train Accuracy = 0.1931 ~ Validation Accuracy = 0.198 Train loss = 2.9447 Validation loss = 2.865
EPOCH 5 ...
Train Accuracy = 0.2779 ~ Validation Accuracy = 0.298 Train loss = 2.5941 Validation loss = 2.452
EPOCH 6 ...
Train Accuracy = 0.3946 ~ Validation Accuracy = 0.421 Train loss = 2.0108 Validation loss = 1.914
EPOCH 7 ...
Train Accuracy = 0.4746 ~ Validation Accuracy = 0.495 Train loss = 1.6699 Validation loss = 1.575
EPOCH 8 ...
Train Accuracy = 0.5911 ~ Validation Accuracy = 0.623 Train loss = 1.2952 Validation loss = 1.217
EPOCH 9 ...
Train Accuracy = 0.7675 ~ Validation Accuracy = 0.807 Train loss = 0.7990 Validation loss = 0.672
EPOCH 10 ...
Train Accuracy = 0.8679 ~ Validation Accuracy = 0.898 Train loss = 0.3831 Validation loss = 0.263
EPOCH 11 ...
Train Accuracy = 0.9475 ~ Validation Accuracy = 0.976 Train loss = 0.1610 Validation loss = 0.074
EPOCH 12 ...
Train Accuracy = 0.9673 ~ Validation Accuracy = 0.986 Train loss = 0.1058 Validation loss = 0.047
EPOCH 13 ...
Train Accuracy = 0.9756 ~ Validation Accuracy = 0.990 Train loss = 0.0816 Validation loss = 0.033
EPOCH 14 ...
Train Accuracy = 0.9836 ~ Validation Accuracy = 0.994 Train loss = 0.0555 Validation loss = 0.020
EPOCH 15 ...
Train Accuracy = 0.9905 ~ Validation Accuracy = 0.996 Train loss = 0.0311 Validation loss = 0.016
EPOCH 16 ...
Train Accuracy = 0.9902 ~ Validation Accuracy = 0.996 Train loss = 0.0329 Validation loss = 0.017
EPOCH 17 ...
Train Accuracy = 0.9910 ~ Validation Accuracy = 0.997 Train loss = 0.0309 Validation loss = 0.013
EPOCH 18 ...
Train Accuracy = 0.9925 ~ Validation Accuracy = 0.997 Train loss = 0.0253 Validation loss = 0.014
EPOCH 19 ...
Train Accuracy = 0.9933 ~ Validation Accuracy = 0.996 Train loss = 0.0232 Validation loss = 0.014
EPOCH 20 ...
Train Accuracy = 0.9900 ~ Validation Accuracy = 0.994 Train loss = 0.0379 Validation loss = 0.026
EPOCH 21 ...
Train Accuracy = 0.9951 ~ Validation Accuracy = 0.998 Train loss = 0.0179 Validation loss = 0.011
EPOCH 22 ...
Train Accuracy = 0.9950 ~ Validation Accuracy = 0.997 Train loss = 0.0168 Validation loss = 0.019
EPOCH 23 ...
Train Accuracy = 0.9955 ~ Validation Accuracy = 0.997 Train loss = 0.0170 Validation loss = 0.021
Early stop, last five validation loss mean is  0.0182051494586
learning rate =  5.66564424791e-05
Training...

EPOCH 1 ...
Train Accuracy = 0.0538 ~ Validation Accuracy = 0.048 Train loss = 3.7283 Validation loss = 3.733
EPOCH 2 ...
Train Accuracy = 0.0677 ~ Validation Accuracy = 0.070 Train loss = 3.6328 Validation loss = 3.622
EPOCH 3 ...
Train Accuracy = 0.0739 ~ Validation Accuracy = 0.081 Train loss = 3.5701 Validation loss = 3.548
EPOCH 4 ...
Train Accuracy = 0.0974 ~ Validation Accuracy = 0.109 Train loss = 3.4620 Validation loss = 3.417
EPOCH 5 ...
Train Accuracy = 0.0828 ~ Validation Accuracy = 0.099 Train loss = 3.4853 Validation loss = 3.427
EPOCH 6 ...
Train Accuracy = 0.1117 ~ Validation Accuracy = 0.118 Train loss = 3.3205 Validation loss = 3.279
EPOCH 7 ...
Train Accuracy = 0.1470 ~ Validation Accuracy = 0.176 Train loss = 3.1908 Validation loss = 3.096
EPOCH 8 ...
Train Accuracy = 0.1758 ~ Validation Accuracy = 0.190 Train loss = 3.0847 Validation loss = 3.007
EPOCH 9 ...
Train Accuracy = 0.1762 ~ Validation Accuracy = 0.188 Train loss = 3.0214 Validation loss = 2.917
EPOCH 10 ...
Train Accuracy = 0.2197 ~ Validation Accuracy = 0.247 Train loss = 2.8351 Validation loss = 2.681
EPOCH 11 ...
Train Accuracy = 0.2527 ~ Validation Accuracy = 0.284 Train loss = 2.6871 Validation loss = 2.548
EPOCH 12 ...
Train Accuracy = 0.2770 ~ Validation Accuracy = 0.316 Train loss = 2.5480 Validation loss = 2.431
EPOCH 13 ...
Train Accuracy = 0.3282 ~ Validation Accuracy = 0.356 Train loss = 2.4141 Validation loss = 2.304
EPOCH 14 ...
Train Accuracy = 0.3454 ~ Validation Accuracy = 0.370 Train loss = 2.3047 Validation loss = 2.191
EPOCH 15 ...
Train Accuracy = 0.3578 ~ Validation Accuracy = 0.372 Train loss = 2.1236 Validation loss = 2.003
EPOCH 16 ...
Train Accuracy = 0.3645 ~ Validation Accuracy = 0.373 Train loss = 2.0428 Validation loss = 1.948
EPOCH 17 ...
Train Accuracy = 0.4156 ~ Validation Accuracy = 0.427 Train loss = 1.9523 Validation loss = 1.880
EPOCH 18 ...
Train Accuracy = 0.4259 ~ Validation Accuracy = 0.434 Train loss = 1.8958 Validation loss = 1.831
EPOCH 19 ...
Train Accuracy = 0.4466 ~ Validation Accuracy = 0.467 Train loss = 1.8422 Validation loss = 1.781
EPOCH 20 ...
Train Accuracy = 0.4640 ~ Validation Accuracy = 0.490 Train loss = 1.7703 Validation loss = 1.663
EPOCH 21 ...
Train Accuracy = 0.5307 ~ Validation Accuracy = 0.564 Train loss = 1.5403 Validation loss = 1.366
EPOCH 22 ...
Train Accuracy = 0.5678 ~ Validation Accuracy = 0.595 Train loss = 1.3562 Validation loss = 1.205
EPOCH 23 ...
Train Accuracy = 0.6216 ~ Validation Accuracy = 0.658 Train loss = 1.1913 Validation loss = 1.030
EPOCH 24 ...
Train Accuracy = 0.6523 ~ Validation Accuracy = 0.693 Train loss = 1.0859 Validation loss = 0.916
EPOCH 25 ...
Train Accuracy = 0.6925 ~ Validation Accuracy = 0.745 Train loss = 0.9877 Validation loss = 0.817
EPOCH 26 ...
Train Accuracy = 0.7537 ~ Validation Accuracy = 0.799 Train loss = 0.8232 Validation loss = 0.647
EPOCH 27 ...
Train Accuracy = 0.7838 ~ Validation Accuracy = 0.830 Train loss = 0.7130 Validation loss = 0.538
EPOCH 28 ...
Train Accuracy = 0.8036 ~ Validation Accuracy = 0.843 Train loss = 0.6325 Validation loss = 0.479
EPOCH 29 ...
Train Accuracy = 0.8248 ~ Validation Accuracy = 0.862 Train loss = 0.5602 Validation loss = 0.416
EPOCH 30 ...
Train Accuracy = 0.8432 ~ Validation Accuracy = 0.876 Train loss = 0.4883 Validation loss = 0.354
EPOCH 31 ...
Train Accuracy = 0.8626 ~ Validation Accuracy = 0.901 Train loss = 0.4268 Validation loss = 0.286
EPOCH 32 ...
Train Accuracy = 0.8870 ~ Validation Accuracy = 0.924 Train loss = 0.3464 Validation loss = 0.223
EPOCH 33 ...
Train Accuracy = 0.9045 ~ Validation Accuracy = 0.938 Train loss = 0.2970 Validation loss = 0.184
EPOCH 34 ...
Train Accuracy = 0.9206 ~ Validation Accuracy = 0.954 Train loss = 0.2476 Validation loss = 0.148
EPOCH 35 ...
Train Accuracy = 0.9306 ~ Validation Accuracy = 0.961 Train loss = 0.2102 Validation loss = 0.123
EPOCH 36 ...
Train Accuracy = 0.9415 ~ Validation Accuracy = 0.970 Train loss = 0.1835 Validation loss = 0.101
EPOCH 37 ...
Train Accuracy = 0.9513 ~ Validation Accuracy = 0.977 Train loss = 0.1566 Validation loss = 0.081
EPOCH 38 ...
Train Accuracy = 0.9572 ~ Validation Accuracy = 0.979 Train loss = 0.1380 Validation loss = 0.073
EPOCH 39 ...
Train Accuracy = 0.9585 ~ Validation Accuracy = 0.982 Train loss = 0.1290 Validation loss = 0.065
EPOCH 40 ...
Train Accuracy = 0.9683 ~ Validation Accuracy = 0.984 Train loss = 0.1039 Validation loss = 0.053
EPOCH 41 ...
Train Accuracy = 0.9756 ~ Validation Accuracy = 0.989 Train loss = 0.0814 Validation loss = 0.035
EPOCH 42 ...
Train Accuracy = 0.9774 ~ Validation Accuracy = 0.990 Train loss = 0.0742 Validation loss = 0.037
EPOCH 43 ...
Train Accuracy = 0.9784 ~ Validation Accuracy = 0.990 Train loss = 0.0707 Validation loss = 0.037
EPOCH 44 ...
Train Accuracy = 0.9834 ~ Validation Accuracy = 0.993 Train loss = 0.0553 Validation loss = 0.026
EPOCH 45 ...
Train Accuracy = 0.9834 ~ Validation Accuracy = 0.992 Train loss = 0.0524 Validation loss = 0.030
EPOCH 46 ...
Train Accuracy = 0.9861 ~ Validation Accuracy = 0.994 Train loss = 0.0441 Validation loss = 0.026
EPOCH 47 ...
Train Accuracy = 0.9876 ~ Validation Accuracy = 0.993 Train loss = 0.0402 Validation loss = 0.024
EPOCH 48 ...
Train Accuracy = 0.9885 ~ Validation Accuracy = 0.994 Train loss = 0.0375 Validation loss = 0.020
EPOCH 49 ...
Train Accuracy = 0.9887 ~ Validation Accuracy = 0.994 Train loss = 0.0352 Validation loss = 0.022
EPOCH 50 ...
Train Accuracy = 0.9904 ~ Validation Accuracy = 0.994 Train loss = 0.0305 Validation loss = 0.024
learning rate =  0.000974566991724
Training...

EPOCH 1 ...
Train Accuracy = 0.0659 ~ Validation Accuracy = 0.077 Train loss = 3.4632 Validation loss = 3.423
EPOCH 2 ...
Train Accuracy = 0.0575 ~ Validation Accuracy = 0.057 Train loss = 3.4867 Validation loss = 3.483
EPOCH 3 ...
Train Accuracy = 0.0575 ~ Validation Accuracy = 0.057 Train loss = 3.4875 Validation loss = 3.483
EPOCH 4 ...
Train Accuracy = 0.0575 ~ Validation Accuracy = 0.057 Train loss = 3.4865 Validation loss = 3.483
EPOCH 5 ...
Train Accuracy = 0.0575 ~ Validation Accuracy = 0.057 Train loss = 3.4865 Validation loss = 3.483
EPOCH 6 ...
Train Accuracy = 0.0575 ~ Validation Accuracy = 0.057 Train loss = 3.4865 Validation loss = 3.483
EPOCH 7 ...
Train Accuracy = 0.0575 ~ Validation Accuracy = 0.057 Train loss = 3.4866 Validation loss = 3.483
EPOCH 8 ...
Train Accuracy = 0.0575 ~ Validation Accuracy = 0.057 Train loss = 3.4865 Validation loss = 3.483
Early stop, last five validation loss mean is  3.48315064616
Model saved
learning rate =  0.000516106525701
Training...

EPOCH 1 ...
Train Accuracy = 0.0969 ~ Validation Accuracy = 0.121 Train loss = 3.3825 Validation loss = 3.288
EPOCH 2 ...
Train Accuracy = 0.1710 ~ Validation Accuracy = 0.188 Train loss = 2.9548 Validation loss = 2.854
EPOCH 3 ...
Train Accuracy = 0.2232 ~ Validation Accuracy = 0.240 Train loss = 2.7629 Validation loss = 2.676
EPOCH 4 ...
Train Accuracy = 0.2866 ~ Validation Accuracy = 0.310 Train loss = 2.4578 Validation loss = 2.334
EPOCH 5 ...
Train Accuracy = 0.2958 ~ Validation Accuracy = 0.292 Train loss = 2.1685 Validation loss = 2.092
EPOCH 6 ...
Train Accuracy = 0.4380 ~ Validation Accuracy = 0.452 Train loss = 1.7204 Validation loss = 1.644
EPOCH 7 ...
Train Accuracy = 0.6485 ~ Validation Accuracy = 0.684 Train loss = 1.0701 Validation loss = 0.922
EPOCH 8 ...
Train Accuracy = 0.8501 ~ Validation Accuracy = 0.907 Train loss = 0.4856 Validation loss = 0.308
EPOCH 9 ...
Train Accuracy = 0.9443 ~ Validation Accuracy = 0.973 Train loss = 0.1812 Validation loss = 0.085
EPOCH 10 ...
Train Accuracy = 0.9623 ~ Validation Accuracy = 0.985 Train loss = 0.1208 Validation loss = 0.047
EPOCH 11 ...
Train Accuracy = 0.9802 ~ Validation Accuracy = 0.994 Train loss = 0.0646 Validation loss = 0.023
EPOCH 12 ...
Train Accuracy = 0.9824 ~ Validation Accuracy = 0.993 Train loss = 0.0580 Validation loss = 0.022
EPOCH 13 ...
Train Accuracy = 0.9829 ~ Validation Accuracy = 0.994 Train loss = 0.0570 Validation loss = 0.018
EPOCH 14 ...
Train Accuracy = 0.9871 ~ Validation Accuracy = 0.997 Train loss = 0.0441 Validation loss = 0.015
EPOCH 15 ...
Train Accuracy = 0.9893 ~ Validation Accuracy = 0.996 Train loss = 0.0387 Validation loss = 0.019
EPOCH 16 ...
Train Accuracy = 0.9925 ~ Validation Accuracy = 0.996 Train loss = 0.0242 Validation loss = 0.019
EPOCH 17 ...
Train Accuracy = 0.9921 ~ Validation Accuracy = 0.996 Train loss = 0.0261 Validation loss = 0.014
EPOCH 18 ...
Train Accuracy = 0.9899 ~ Validation Accuracy = 0.997 Train loss = 0.0363 Validation loss = 0.022
EPOCH 19 ...
Train Accuracy = 0.9936 ~ Validation Accuracy = 0.996 Train loss = 0.0239 Validation loss = 0.016
EPOCH 20 ...
Train Accuracy = 0.9930 ~ Validation Accuracy = 0.997 Train loss = 0.0256 Validation loss = 0.019
EPOCH 21 ...
Train Accuracy = 0.9917 ~ Validation Accuracy = 0.996 Train loss = 0.0384 Validation loss = 0.022
Early stop, last five validation loss mean is  0.0185091147489
learning rate =  2.97530451143e-05
Training...

EPOCH 1 ...
Train Accuracy = 0.0544 ~ Validation Accuracy = 0.056 Train loss = 3.8723 Validation loss = 3.873
EPOCH 2 ...
Train Accuracy = 0.0546 ~ Validation Accuracy = 0.056 Train loss = 3.7213 Validation loss = 3.722
EPOCH 3 ...
Train Accuracy = 0.0578 ~ Validation Accuracy = 0.057 Train loss = 3.6811 Validation loss = 3.683
EPOCH 4 ...
Train Accuracy = 0.0577 ~ Validation Accuracy = 0.057 Train loss = 3.6549 Validation loss = 3.656
EPOCH 5 ...
Train Accuracy = 0.0576 ~ Validation Accuracy = 0.057 Train loss = 3.6327 Validation loss = 3.633
EPOCH 6 ...
Train Accuracy = 0.0575 ~ Validation Accuracy = 0.057 Train loss = 3.6137 Validation loss = 3.615
EPOCH 7 ...
Train Accuracy = 0.0575 ~ Validation Accuracy = 0.057 Train loss = 3.5960 Validation loss = 3.595
EPOCH 8 ...
Train Accuracy = 0.0575 ~ Validation Accuracy = 0.057 Train loss = 3.5808 Validation loss = 3.580
EPOCH 9 ...
Train Accuracy = 0.0575 ~ Validation Accuracy = 0.057 Train loss = 3.5679 Validation loss = 3.567
EPOCH 10 ...
Train Accuracy = 0.0575 ~ Validation Accuracy = 0.057 Train loss = 3.5555 Validation loss = 3.554
EPOCH 11 ...
Train Accuracy = 0.0575 ~ Validation Accuracy = 0.057 Train loss = 3.5449 Validation loss = 3.543
EPOCH 12 ...
Train Accuracy = 0.0575 ~ Validation Accuracy = 0.057 Train loss = 3.5360 Validation loss = 3.534
EPOCH 13 ...
Train Accuracy = 0.0575 ~ Validation Accuracy = 0.057 Train loss = 3.5283 Validation loss = 3.526
EPOCH 14 ...
Train Accuracy = 0.0575 ~ Validation Accuracy = 0.057 Train loss = 3.5217 Validation loss = 3.520
EPOCH 15 ...
Train Accuracy = 0.0575 ~ Validation Accuracy = 0.057 Train loss = 3.5161 Validation loss = 3.514
EPOCH 16 ...
Train Accuracy = 0.0575 ~ Validation Accuracy = 0.057 Train loss = 3.5114 Validation loss = 3.509
EPOCH 17 ...
Train Accuracy = 0.0575 ~ Validation Accuracy = 0.057 Train loss = 3.5074 Validation loss = 3.505
EPOCH 18 ...
Train Accuracy = 0.0575 ~ Validation Accuracy = 0.057 Train loss = 3.5041 Validation loss = 3.502
EPOCH 19 ...
Train Accuracy = 0.0575 ~ Validation Accuracy = 0.057 Train loss = 3.5013 Validation loss = 3.499
EPOCH 20 ...
Train Accuracy = 0.0575 ~ Validation Accuracy = 0.057 Train loss = 3.4989 Validation loss = 3.496
EPOCH 21 ...
Train Accuracy = 0.0575 ~ Validation Accuracy = 0.057 Train loss = 3.4970 Validation loss = 3.494
EPOCH 22 ...
Train Accuracy = 0.0575 ~ Validation Accuracy = 0.057 Train loss = 3.4953 Validation loss = 3.493
EPOCH 23 ...
Train Accuracy = 0.0575 ~ Validation Accuracy = 0.057 Train loss = 3.4940 Validation loss = 3.491
EPOCH 24 ...
Train Accuracy = 0.0575 ~ Validation Accuracy = 0.057 Train loss = 3.4928 Validation loss = 3.490
EPOCH 25 ...
Train Accuracy = 0.0575 ~ Validation Accuracy = 0.057 Train loss = 3.4919 Validation loss = 3.489
EPOCH 26 ...
Train Accuracy = 0.0575 ~ Validation Accuracy = 0.057 Train loss = 3.4911 Validation loss = 3.488
EPOCH 27 ...
Train Accuracy = 0.0575 ~ Validation Accuracy = 0.057 Train loss = 3.4904 Validation loss = 3.487
EPOCH 28 ...
Train Accuracy = 0.0575 ~ Validation Accuracy = 0.057 Train loss = 3.4898 Validation loss = 3.487
EPOCH 29 ...
Train Accuracy = 0.0575 ~ Validation Accuracy = 0.057 Train loss = 3.4893 Validation loss = 3.486
EPOCH 30 ...
Train Accuracy = 0.0575 ~ Validation Accuracy = 0.057 Train loss = 3.4889 Validation loss = 3.486
EPOCH 31 ...
Train Accuracy = 0.0575 ~ Validation Accuracy = 0.057 Train loss = 3.4886 Validation loss = 3.485
EPOCH 32 ...
Train Accuracy = 0.0575 ~ Validation Accuracy = 0.057 Train loss = 3.4883 Validation loss = 3.485
EPOCH 33 ...
Train Accuracy = 0.0575 ~ Validation Accuracy = 0.057 Train loss = 3.4880 Validation loss = 3.485
EPOCH 34 ...
Train Accuracy = 0.0575 ~ Validation Accuracy = 0.057 Train loss = 3.4878 Validation loss = 3.485
EPOCH 35 ...
Train Accuracy = 0.0575 ~ Validation Accuracy = 0.057 Train loss = 3.4877 Validation loss = 3.484
EPOCH 36 ...
Train Accuracy = 0.0575 ~ Validation Accuracy = 0.057 Train loss = 3.4875 Validation loss = 3.484
EPOCH 37 ...
Train Accuracy = 0.0575 ~ Validation Accuracy = 0.057 Train loss = 3.4874 Validation loss = 3.484
EPOCH 38 ...
Train Accuracy = 0.0575 ~ Validation Accuracy = 0.057 Train loss = 3.4872 Validation loss = 3.484
EPOCH 39 ...
Train Accuracy = 0.0575 ~ Validation Accuracy = 0.057 Train loss = 3.4872 Validation loss = 3.484
EPOCH 40 ...
Train Accuracy = 0.0575 ~ Validation Accuracy = 0.057 Train loss = 3.4871 Validation loss = 3.484
EPOCH 41 ...
Train Accuracy = 0.0575 ~ Validation Accuracy = 0.057 Train loss = 3.4870 Validation loss = 3.484
EPOCH 42 ...
Train Accuracy = 0.0575 ~ Validation Accuracy = 0.057 Train loss = 3.4869 Validation loss = 3.484
EPOCH 43 ...
Train Accuracy = 0.0575 ~ Validation Accuracy = 0.057 Train loss = 3.4869 Validation loss = 3.484
EPOCH 44 ...
Train Accuracy = 0.0575 ~ Validation Accuracy = 0.057 Train loss = 3.4868 Validation loss = 3.484
EPOCH 45 ...
Train Accuracy = 0.0575 ~ Validation Accuracy = 0.057 Train loss = 3.4868 Validation loss = 3.484
EPOCH 46 ...
Train Accuracy = 0.0575 ~ Validation Accuracy = 0.057 Train loss = 3.4868 Validation loss = 3.483
EPOCH 47 ...
Train Accuracy = 0.0575 ~ Validation Accuracy = 0.057 Train loss = 3.4867 Validation loss = 3.483
EPOCH 48 ...
Train Accuracy = 0.0575 ~ Validation Accuracy = 0.057 Train loss = 3.4867 Validation loss = 3.483
EPOCH 49 ...
Train Accuracy = 0.0575 ~ Validation Accuracy = 0.057 Train loss = 3.4867 Validation loss = 3.483
EPOCH 50 ...
Train Accuracy = 0.0575 ~ Validation Accuracy = 0.057 Train loss = 3.4866 Validation loss = 3.483
Model saved
learning rate =  0.000102637939021
Training...

EPOCH 1 ...
Train Accuracy = 0.0586 ~ Validation Accuracy = 0.057 Train loss = 3.6745 Validation loss = 3.675
EPOCH 2 ...
Train Accuracy = 0.0593 ~ Validation Accuracy = 0.058 Train loss = 3.5995 Validation loss = 3.602
EPOCH 3 ...
Train Accuracy = 0.0838 ~ Validation Accuracy = 0.086 Train loss = 3.4265 Validation loss = 3.402
EPOCH 4 ...
Train Accuracy = 0.0575 ~ Validation Accuracy = 0.057 Train loss = 3.5291 Validation loss = 3.527
EPOCH 5 ...
Train Accuracy = 0.1018 ~ Validation Accuracy = 0.117 Train loss = 3.2783 Validation loss = 3.232
EPOCH 6 ...
Train Accuracy = 0.1148 ~ Validation Accuracy = 0.106 Train loss = 3.2130 Validation loss = 3.170
EPOCH 7 ...
Train Accuracy = 0.1155 ~ Validation Accuracy = 0.118 Train loss = 3.2135 Validation loss = 3.158
EPOCH 8 ...
Train Accuracy = 0.1524 ~ Validation Accuracy = 0.164 Train loss = 3.0780 Validation loss = 3.010
EPOCH 9 ...
Train Accuracy = 0.1805 ~ Validation Accuracy = 0.202 Train loss = 2.9968 Validation loss = 2.915
EPOCH 10 ...
Train Accuracy = 0.2594 ~ Validation Accuracy = 0.288 Train loss = 2.7588 Validation loss = 2.652
EPOCH 11 ...
Train Accuracy = 0.3024 ~ Validation Accuracy = 0.332 Train loss = 2.4969 Validation loss = 2.382
EPOCH 12 ...
Train Accuracy = 0.3280 ~ Validation Accuracy = 0.344 Train loss = 2.3186 Validation loss = 2.204
EPOCH 13 ...
Train Accuracy = 0.3949 ~ Validation Accuracy = 0.408 Train loss = 2.0470 Validation loss = 1.921
EPOCH 14 ...
Train Accuracy = 0.4191 ~ Validation Accuracy = 0.424 Train loss = 1.8170 Validation loss = 1.690
EPOCH 15 ...
Train Accuracy = 0.4995 ~ Validation Accuracy = 0.514 Train loss = 1.6097 Validation loss = 1.474
EPOCH 16 ...
Train Accuracy = 0.5249 ~ Validation Accuracy = 0.550 Train loss = 1.4409 Validation loss = 1.306
EPOCH 17 ...
Train Accuracy = 0.5778 ~ Validation Accuracy = 0.613 Train loss = 1.2319 Validation loss = 1.078
EPOCH 18 ...
Train Accuracy = 0.6390 ~ Validation Accuracy = 0.686 Train loss = 1.0052 Validation loss = 0.832
EPOCH 19 ...
Train Accuracy = 0.7067 ~ Validation Accuracy = 0.763 Train loss = 0.8403 Validation loss = 0.650
EPOCH 20 ...
Train Accuracy = 0.7894 ~ Validation Accuracy = 0.847 Train loss = 0.6254 Validation loss = 0.452
EPOCH 21 ...
Train Accuracy = 0.8414 ~ Validation Accuracy = 0.883 Train loss = 0.4769 Validation loss = 0.327
EPOCH 22 ...
Train Accuracy = 0.8756 ~ Validation Accuracy = 0.916 Train loss = 0.3744 Validation loss = 0.238
EPOCH 23 ...
Train Accuracy = 0.9122 ~ Validation Accuracy = 0.951 Train loss = 0.2740 Validation loss = 0.150
EPOCH 24 ...
Train Accuracy = 0.9350 ~ Validation Accuracy = 0.969 Train loss = 0.2011 Validation loss = 0.096
EPOCH 25 ...
Train Accuracy = 0.9498 ~ Validation Accuracy = 0.973 Train loss = 0.1590 Validation loss = 0.080
EPOCH 26 ...
Train Accuracy = 0.9599 ~ Validation Accuracy = 0.983 Train loss = 0.1252 Validation loss = 0.056
EPOCH 27 ...
Train Accuracy = 0.9702 ~ Validation Accuracy = 0.989 Train loss = 0.0961 Validation loss = 0.042
EPOCH 28 ...
Train Accuracy = 0.9773 ~ Validation Accuracy = 0.990 Train loss = 0.0727 Validation loss = 0.030
EPOCH 29 ...
Train Accuracy = 0.9832 ~ Validation Accuracy = 0.991 Train loss = 0.0537 Validation loss = 0.025
EPOCH 30 ...
Train Accuracy = 0.9830 ~ Validation Accuracy = 0.992 Train loss = 0.0549 Validation loss = 0.025
EPOCH 31 ...
Train Accuracy = 0.9875 ~ Validation Accuracy = 0.996 Train loss = 0.0398 Validation loss = 0.012
EPOCH 32 ...
Train Accuracy = 0.9907 ~ Validation Accuracy = 0.995 Train loss = 0.0293 Validation loss = 0.017
EPOCH 33 ...
Train Accuracy = 0.9916 ~ Validation Accuracy = 0.995 Train loss = 0.0269 Validation loss = 0.014
EPOCH 34 ...
Train Accuracy = 0.9928 ~ Validation Accuracy = 0.996 Train loss = 0.0229 Validation loss = 0.015
EPOCH 35 ...
Train Accuracy = 0.9909 ~ Validation Accuracy = 0.995 Train loss = 0.0289 Validation loss = 0.017
EPOCH 36 ...
Train Accuracy = 0.9950 ~ Validation Accuracy = 0.997 Train loss = 0.0160 Validation loss = 0.013
EPOCH 37 ...
Train Accuracy = 0.9956 ~ Validation Accuracy = 0.996 Train loss = 0.0139 Validation loss = 0.012
EPOCH 38 ...
Train Accuracy = 0.9955 ~ Validation Accuracy = 0.997 Train loss = 0.0148 Validation loss = 0.012
EPOCH 39 ...
Train Accuracy = 0.9967 ~ Validation Accuracy = 0.997 Train loss = 0.0106 Validation loss = 0.010
EPOCH 40 ...
Train Accuracy = 0.9957 ~ Validation Accuracy = 0.998 Train loss = 0.0135 Validation loss = 0.009
EPOCH 41 ...
Train Accuracy = 0.9958 ~ Validation Accuracy = 0.998 Train loss = 0.0136 Validation loss = 0.008
EPOCH 42 ...
Train Accuracy = 0.9966 ~ Validation Accuracy = 0.997 Train loss = 0.0108 Validation loss = 0.010
EPOCH 43 ...
Train Accuracy = 0.9958 ~ Validation Accuracy = 0.997 Train loss = 0.0133 Validation loss = 0.013
EPOCH 44 ...
Train Accuracy = 0.9981 ~ Validation Accuracy = 0.998 Train loss = 0.0067 Validation loss = 0.007
EPOCH 45 ...
Train Accuracy = 0.9979 ~ Validation Accuracy = 0.997 Train loss = 0.0065 Validation loss = 0.013
EPOCH 46 ...
Train Accuracy = 0.9981 ~ Validation Accuracy = 0.998 Train loss = 0.0063 Validation loss = 0.008
EPOCH 47 ...
Train Accuracy = 0.9979 ~ Validation Accuracy = 0.998 Train loss = 0.0072 Validation loss = 0.013
EPOCH 48 ...
Train Accuracy = 0.9985 ~ Validation Accuracy = 0.998 Train loss = 0.0051 Validation loss = 0.008
EPOCH 49 ...
Train Accuracy = 0.9972 ~ Validation Accuracy = 0.997 Train loss = 0.0088 Validation loss = 0.012
EPOCH 50 ...
Train Accuracy = 0.9990 ~ Validation Accuracy = 0.998 Train loss = 0.0033 Validation loss = 0.006
learning rate =  0.000119830023936
Training...

EPOCH 1 ...
Train Accuracy = 0.0578 ~ Validation Accuracy = 0.057 Train loss = 3.6612 Validation loss = 3.660
EPOCH 2 ...
Train Accuracy = 0.0576 ~ Validation Accuracy = 0.057 Train loss = 3.5855 Validation loss = 3.585
EPOCH 3 ...
Train Accuracy = 0.0575 ~ Validation Accuracy = 0.057 Train loss = 3.5407 Validation loss = 3.542
EPOCH 4 ...
Train Accuracy = 0.0575 ~ Validation Accuracy = 0.057 Train loss = 3.5148 Validation loss = 3.512
EPOCH 5 ...
Train Accuracy = 0.0575 ~ Validation Accuracy = 0.057 Train loss = 3.5008 Validation loss = 3.498
EPOCH 6 ...
Train Accuracy = 0.0575 ~ Validation Accuracy = 0.057 Train loss = 3.4938 Validation loss = 3.491
EPOCH 7 ...
Train Accuracy = 0.0575 ~ Validation Accuracy = 0.057 Train loss = 3.4903 Validation loss = 3.487
EPOCH 8 ...
Train Accuracy = 0.0575 ~ Validation Accuracy = 0.057 Train loss = 3.4886 Validation loss = 3.485
EPOCH 9 ...
Train Accuracy = 0.0575 ~ Validation Accuracy = 0.057 Train loss = 3.4876 Validation loss = 3.484
EPOCH 10 ...
Train Accuracy = 0.0575 ~ Validation Accuracy = 0.057 Train loss = 3.4871 Validation loss = 3.484
EPOCH 11 ...
Train Accuracy = 0.0575 ~ Validation Accuracy = 0.057 Train loss = 3.4869 Validation loss = 3.484
EPOCH 12 ...
Train Accuracy = 0.0575 ~ Validation Accuracy = 0.057 Train loss = 3.4867 Validation loss = 3.483
EPOCH 13 ...
Train Accuracy = 0.0575 ~ Validation Accuracy = 0.057 Train loss = 3.4866 Validation loss = 3.483
EPOCH 14 ...
Train Accuracy = 0.0575 ~ Validation Accuracy = 0.057 Train loss = 3.4866 Validation loss = 3.483
EPOCH 15 ...
Train Accuracy = 0.0575 ~ Validation Accuracy = 0.057 Train loss = 3.4865 Validation loss = 3.483
EPOCH 16 ...
Train Accuracy = 0.0575 ~ Validation Accuracy = 0.057 Train loss = 3.4865 Validation loss = 3.483
EPOCH 17 ...
Train Accuracy = 0.0575 ~ Validation Accuracy = 0.057 Train loss = 3.4865 Validation loss = 3.483
EPOCH 18 ...
Train Accuracy = 0.0575 ~ Validation Accuracy = 0.057 Train loss = 3.4865 Validation loss = 3.483
EPOCH 19 ...
Train Accuracy = 0.0575 ~ Validation Accuracy = 0.057 Train loss = 3.4865 Validation loss = 3.483
EPOCH 20 ...
Train Accuracy = 0.0575 ~ Validation Accuracy = 0.057 Train loss = 3.4865 Validation loss = 3.483
In [ ]:
# Good result    8.89639749842e-05  6.23291123951e-05

# Potential 2.06207481419e-05 3.17401106699e-05 1.72062777121e-05   7.70175254958e-06(too slow) 6.12041999144e-06
In [49]:
x = tf.placeholder(tf.float32, (None, 32, 32, 3))
y = tf.placeholder(tf.int32, (None))
one_hot_y = tf.one_hot(y, 43)
keep_prob = tf.placeholder(tf.float32)

rate =7.3951e-05
print('learning rate = ', rate)
logits = conv_net(x,keep_prob)
#logits = LeNet(x,keep_prob)
cross_entropy = tf.nn.softmax_cross_entropy_with_logits(logits, one_hot_y)
loss_operation = tf.reduce_mean(cross_entropy)
optimizer = tf.train.AdamOptimizer(learning_rate = rate)
training_operation = optimizer.minimize(loss_operation)


correct_prediction = tf.equal(tf.argmax(logits, 1), tf.argmax(one_hot_y, 1))
accuracy_operation = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
saver = tf.train.Saver()

#Add loss in the evaluate function
def evaluate(X_data, y_data):
    num_examples = len(X_data)
    total_accuracy = 0
    total_loss = 0
    sess = tf.get_default_session()
    for offset in range(0, num_examples, BATCH_SIZE):
        batch_x, batch_y = X_data[offset:offset+BATCH_SIZE], y_data[offset:offset+BATCH_SIZE]
        accuracy,_loss = sess.run([accuracy_operation,loss_operation], 
                                  feed_dict={x: batch_x, y: batch_y,keep_prob: 1.0})
        total_accuracy += (accuracy * len(batch_x))
        total_loss += (_loss*len(batch_x))
    return(total_accuracy / num_examples, total_loss / num_examples)

EPOCHS = 100
BATCH_SIZE = 64
DROPOUT = 0.8
EARLY_STOP = 5


with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    num_examples = len(X_train)
    print("Training...")
    print()
    train_acc = []
    valid_acc = []
    train_loss = []
    valid_loss = []
    for i in range(EPOCHS):
        X_train, y_train = shuffle(X_train, y_train)
        for offset in range(0, num_examples, BATCH_SIZE):
            end = offset + BATCH_SIZE
            batch_x, batch_y = X_train[offset:end], y_train[offset:end]
            sess.run(training_operation, feed_dict={x: batch_x, y: batch_y,keep_prob:DROPOUT})

        training_accuracy,training_loss   =   evaluate(X_train, y_train)
        validation_accuracy,validation_loss = evaluate(X_validation, y_validation)
        train_acc.append(training_accuracy)
        valid_acc.append(validation_accuracy)
        train_loss.append(training_loss)
        valid_loss.append(validation_loss)
        print("EPOCH {} ...".format(i+1))
        print("Train Accuracy = {:.4f} ~".format(training_accuracy),
              "Validation Accuracy = {:.3f}".format(validation_accuracy),
              "Train loss = {:.4f}".format(training_loss),
              "Validation loss = {:.3f}".format(validation_loss))
        if len(valid_loss)>7 and \
        valid_loss[-1]>= valid_loss[-2] and valid_loss[-1] >= np.mean(valid_loss[-EARLY_STOP-1:]) and\
        valid_loss[-2]>= valid_loss[-3] and valid_loss[-2] >= np.mean(valid_loss[-EARLY_STOP-2:-2]) and\
        valid_loss[-3]>= valid_loss[-4] and valid_loss[-3] >= np.mean(valid_loss[-EARLY_STOP-3:-3]):
            print('Early stop, last five validation loss mean is ',np.mean(valid_loss[-EARLY_STOP:]))
            break

    plt.plot(train_loss, 'b--',valid_loss,'ro')
    plt.show()
    plt.plot(train_acc, 'b--',valid_acc,'ro')
    plt.show()

    saver.save(sess, 'lenet')
    print("Model saved")
learning rate =  7.3951e-05
Training...

EPOCH 1 ...
Train Accuracy = 0.0632 ~ Validation Accuracy = 0.068 Train loss = 3.6365 Validation loss = 3.620
EPOCH 2 ...
Train Accuracy = 0.0661 ~ Validation Accuracy = 0.075 Train loss = 3.5499 Validation loss = 3.519
EPOCH 3 ...
Train Accuracy = 0.1007 ~ Validation Accuracy = 0.113 Train loss = 3.4055 Validation loss = 3.352
EPOCH 4 ...
Train Accuracy = 0.0940 ~ Validation Accuracy = 0.111 Train loss = 3.3949 Validation loss = 3.329
EPOCH 5 ...
Train Accuracy = 0.0808 ~ Validation Accuracy = 0.094 Train loss = 3.4284 Validation loss = 3.377
EPOCH 6 ...
Train Accuracy = 0.1116 ~ Validation Accuracy = 0.139 Train loss = 3.3050 Validation loss = 3.205
EPOCH 7 ...
Train Accuracy = 0.1642 ~ Validation Accuracy = 0.181 Train loss = 3.1280 Validation loss = 3.050
EPOCH 8 ...
Train Accuracy = 0.1811 ~ Validation Accuracy = 0.197 Train loss = 3.0697 Validation loss = 3.004
EPOCH 9 ...
Train Accuracy = 0.1926 ~ Validation Accuracy = 0.201 Train loss = 2.9471 Validation loss = 2.912
EPOCH 10 ...
Train Accuracy = 0.2369 ~ Validation Accuracy = 0.245 Train loss = 2.7503 Validation loss = 2.682
EPOCH 11 ...
Train Accuracy = 0.2550 ~ Validation Accuracy = 0.264 Train loss = 2.6380 Validation loss = 2.589
EPOCH 12 ...
Train Accuracy = 0.3173 ~ Validation Accuracy = 0.344 Train loss = 2.4523 Validation loss = 2.354
EPOCH 13 ...
Train Accuracy = 0.3564 ~ Validation Accuracy = 0.369 Train loss = 2.1391 Validation loss = 2.058
EPOCH 14 ...
Train Accuracy = 0.4142 ~ Validation Accuracy = 0.427 Train loss = 1.9510 Validation loss = 1.858
EPOCH 15 ...
Train Accuracy = 0.4524 ~ Validation Accuracy = 0.453 Train loss = 1.5901 Validation loss = 1.517
EPOCH 16 ...
Train Accuracy = 0.4960 ~ Validation Accuracy = 0.518 Train loss = 1.4001 Validation loss = 1.276
EPOCH 17 ...
Train Accuracy = 0.5631 ~ Validation Accuracy = 0.584 Train loss = 1.1901 Validation loss = 1.060
EPOCH 18 ...
Train Accuracy = 0.6030 ~ Validation Accuracy = 0.633 Train loss = 1.0619 Validation loss = 0.941
EPOCH 19 ...
Train Accuracy = 0.6912 ~ Validation Accuracy = 0.735 Train loss = 0.8990 Validation loss = 0.725
EPOCH 20 ...
Train Accuracy = 0.7682 ~ Validation Accuracy = 0.842 Train loss = 0.7257 Validation loss = 0.544
EPOCH 21 ...
Train Accuracy = 0.8278 ~ Validation Accuracy = 0.884 Train loss = 0.5457 Validation loss = 0.380
EPOCH 22 ...
Train Accuracy = 0.8536 ~ Validation Accuracy = 0.900 Train loss = 0.4632 Validation loss = 0.304
EPOCH 23 ...
Train Accuracy = 0.8889 ~ Validation Accuracy = 0.932 Train loss = 0.3436 Validation loss = 0.211
EPOCH 24 ...
Train Accuracy = 0.9189 ~ Validation Accuracy = 0.957 Train loss = 0.2656 Validation loss = 0.139
EPOCH 25 ...
Train Accuracy = 0.9342 ~ Validation Accuracy = 0.968 Train loss = 0.2153 Validation loss = 0.105
EPOCH 26 ...
Train Accuracy = 0.9500 ~ Validation Accuracy = 0.979 Train loss = 0.1611 Validation loss = 0.073
EPOCH 27 ...
Train Accuracy = 0.9581 ~ Validation Accuracy = 0.985 Train loss = 0.1369 Validation loss = 0.050
EPOCH 28 ...
Train Accuracy = 0.9695 ~ Validation Accuracy = 0.988 Train loss = 0.0948 Validation loss = 0.037
EPOCH 29 ...
Train Accuracy = 0.9741 ~ Validation Accuracy = 0.991 Train loss = 0.0815 Validation loss = 0.034
EPOCH 30 ...
Train Accuracy = 0.9807 ~ Validation Accuracy = 0.993 Train loss = 0.0624 Validation loss = 0.023
EPOCH 31 ...
Train Accuracy = 0.9842 ~ Validation Accuracy = 0.993 Train loss = 0.0506 Validation loss = 0.025
EPOCH 32 ...
Train Accuracy = 0.9871 ~ Validation Accuracy = 0.996 Train loss = 0.0402 Validation loss = 0.014
EPOCH 33 ...
Train Accuracy = 0.9890 ~ Validation Accuracy = 0.995 Train loss = 0.0348 Validation loss = 0.017
EPOCH 34 ...
Train Accuracy = 0.9912 ~ Validation Accuracy = 0.996 Train loss = 0.0282 Validation loss = 0.013
EPOCH 35 ...
Train Accuracy = 0.9920 ~ Validation Accuracy = 0.996 Train loss = 0.0267 Validation loss = 0.024
EPOCH 36 ...
Train Accuracy = 0.9932 ~ Validation Accuracy = 0.996 Train loss = 0.0227 Validation loss = 0.012
EPOCH 37 ...
Train Accuracy = 0.9893 ~ Validation Accuracy = 0.995 Train loss = 0.0343 Validation loss = 0.019
EPOCH 38 ...
Train Accuracy = 0.9948 ~ Validation Accuracy = 0.997 Train loss = 0.0168 Validation loss = 0.010
EPOCH 39 ...
Train Accuracy = 0.9949 ~ Validation Accuracy = 0.997 Train loss = 0.0163 Validation loss = 0.012
EPOCH 40 ...
Train Accuracy = 0.9959 ~ Validation Accuracy = 0.998 Train loss = 0.0135 Validation loss = 0.009
EPOCH 41 ...
Train Accuracy = 0.9963 ~ Validation Accuracy = 0.998 Train loss = 0.0118 Validation loss = 0.009
EPOCH 42 ...
Train Accuracy = 0.9910 ~ Validation Accuracy = 0.997 Train loss = 0.0332 Validation loss = 0.016
EPOCH 43 ...
Train Accuracy = 0.9963 ~ Validation Accuracy = 0.997 Train loss = 0.0115 Validation loss = 0.011
EPOCH 44 ...
Train Accuracy = 0.9978 ~ Validation Accuracy = 0.998 Train loss = 0.0075 Validation loss = 0.008
EPOCH 45 ...
Train Accuracy = 0.9965 ~ Validation Accuracy = 0.997 Train loss = 0.0114 Validation loss = 0.012
EPOCH 46 ...
Train Accuracy = 0.9979 ~ Validation Accuracy = 0.999 Train loss = 0.0065 Validation loss = 0.010
EPOCH 47 ...
Train Accuracy = 0.9974 ~ Validation Accuracy = 0.998 Train loss = 0.0084 Validation loss = 0.009
EPOCH 48 ...
Train Accuracy = 0.9981 ~ Validation Accuracy = 0.998 Train loss = 0.0063 Validation loss = 0.011
EPOCH 49 ...
Train Accuracy = 0.9976 ~ Validation Accuracy = 0.998 Train loss = 0.0074 Validation loss = 0.008
EPOCH 50 ...
Train Accuracy = 0.9982 ~ Validation Accuracy = 0.999 Train loss = 0.0059 Validation loss = 0.004
EPOCH 51 ...
Train Accuracy = 0.9983 ~ Validation Accuracy = 0.999 Train loss = 0.0053 Validation loss = 0.006
EPOCH 52 ...
Train Accuracy = 0.9978 ~ Validation Accuracy = 0.998 Train loss = 0.0074 Validation loss = 0.010
EPOCH 53 ...
Train Accuracy = 0.9984 ~ Validation Accuracy = 0.998 Train loss = 0.0051 Validation loss = 0.007
EPOCH 54 ...
Train Accuracy = 0.9979 ~ Validation Accuracy = 0.999 Train loss = 0.0081 Validation loss = 0.008
EPOCH 55 ...
Train Accuracy = 0.9988 ~ Validation Accuracy = 0.999 Train loss = 0.0040 Validation loss = 0.008
EPOCH 56 ...
Train Accuracy = 0.9985 ~ Validation Accuracy = 0.999 Train loss = 0.0048 Validation loss = 0.006
EPOCH 57 ...
Train Accuracy = 0.9979 ~ Validation Accuracy = 0.999 Train loss = 0.0068 Validation loss = 0.008
EPOCH 58 ...
Train Accuracy = 0.9987 ~ Validation Accuracy = 0.999 Train loss = 0.0044 Validation loss = 0.009
EPOCH 59 ...
Train Accuracy = 0.9985 ~ Validation Accuracy = 0.999 Train loss = 0.0049 Validation loss = 0.008
EPOCH 60 ...
Train Accuracy = 0.9974 ~ Validation Accuracy = 0.999 Train loss = 0.0096 Validation loss = 0.009
EPOCH 61 ...
Train Accuracy = 0.9991 ~ Validation Accuracy = 0.999 Train loss = 0.0030 Validation loss = 0.006
EPOCH 62 ...
Train Accuracy = 0.9995 ~ Validation Accuracy = 0.999 Train loss = 0.0017 Validation loss = 0.006
EPOCH 63 ...
Train Accuracy = 0.9989 ~ Validation Accuracy = 0.999 Train loss = 0.0041 Validation loss = 0.007
EPOCH 64 ...
Train Accuracy = 0.9986 ~ Validation Accuracy = 0.999 Train loss = 0.0047 Validation loss = 0.006
EPOCH 65 ...
Train Accuracy = 0.9980 ~ Validation Accuracy = 0.999 Train loss = 0.0083 Validation loss = 0.010
EPOCH 66 ...
Train Accuracy = 0.9994 ~ Validation Accuracy = 0.999 Train loss = 0.0022 Validation loss = 0.009
EPOCH 67 ...
Train Accuracy = 0.9993 ~ Validation Accuracy = 0.999 Train loss = 0.0026 Validation loss = 0.008
EPOCH 68 ...
Train Accuracy = 0.9976 ~ Validation Accuracy = 0.999 Train loss = 0.0078 Validation loss = 0.005
EPOCH 69 ...
Train Accuracy = 0.9990 ~ Validation Accuracy = 0.999 Train loss = 0.0039 Validation loss = 0.010
EPOCH 70 ...
Train Accuracy = 0.9989 ~ Validation Accuracy = 0.999 Train loss = 0.0036 Validation loss = 0.005
EPOCH 71 ...
Train Accuracy = 0.9990 ~ Validation Accuracy = 0.999 Train loss = 0.0036 Validation loss = 0.005
EPOCH 72 ...
Train Accuracy = 0.9991 ~ Validation Accuracy = 0.999 Train loss = 0.0027 Validation loss = 0.008
EPOCH 73 ...
Train Accuracy = 0.9994 ~ Validation Accuracy = 0.999 Train loss = 0.0020 Validation loss = 0.008
EPOCH 74 ...
Train Accuracy = 0.9987 ~ Validation Accuracy = 0.999 Train loss = 0.0045 Validation loss = 0.007
EPOCH 75 ...
Train Accuracy = 0.9993 ~ Validation Accuracy = 0.999 Train loss = 0.0027 Validation loss = 0.005
EPOCH 76 ...
Train Accuracy = 0.9990 ~ Validation Accuracy = 0.999 Train loss = 0.0036 Validation loss = 0.008
EPOCH 77 ...
Train Accuracy = 0.9995 ~ Validation Accuracy = 0.999 Train loss = 0.0018 Validation loss = 0.008
EPOCH 78 ...
Train Accuracy = 0.9990 ~ Validation Accuracy = 0.997 Train loss = 0.0035 Validation loss = 0.016
Early stop, last five validation loss mean is  0.00885795126682
Model saved
In [50]:
start = 15
plt.plot(train_acc[start:], 'b-',valid_acc[start:],'r-')
plt.savefig('b_size='+str(BATCH_SIZE)+'learn_rate='+str(rate),format='png')
plt.show()

plt.plot(train_loss[start:], 'b-',valid_loss[start:],'r-')
plt.savefig('b_size='+str(BATCH_SIZE)+'learn_rate='+str(rate)+'loss',format='png')
plt.show()
In [36]:
with tf.Session() as sess:
    saver.restore(sess, tf.train.latest_checkpoint('.'))

    test_accuracy = evaluate(X_test, y_test)
    print(test_accuracy)
(0.96991290570438049, 0.44048884815860151)
In [51]:
with tf.Session() as sess:
    saver.restore(sess, tf.train.latest_checkpoint('.'))

    test_accuracy = evaluate(X_test, y_test)
    print(test_accuracy)
(0.9722090262037747, 0.43345779527383183)
In [ ]:
prediction = tf.argmax(logits, 1)

Question 1

Describe how you preprocessed the data. Why did you choose that technique?

Answer:

Preprocessing method

The preprocessing is to find the invariance and symmetry of images in general and at the time try to avoid customized feature engineering to the specific dataset. In the image above, we can clearly seen that there are several universal invariance amount contrast, size, shape.

  1. CLAHE Transform
  2. Data augmentation

http://people.idsia.ch/~juergen/ijcnn2011.pdf In this paper it shows that the CLAHE Transform did the best among other normalization methods, intuitively it make sense because the high variance of the contrast situation in the original dataset. I'll show the transformed images

In [37]:
### Generate data additional data (OPTIONAL!)
### and split the data into training/validation/testing sets here.
### Feel free to use as many code cells as needed.

Question 2

Describe how you set up the training, validation and testing data for your model. Optional: If you generated additional data, how did you generate the data? Why did you generate the data? What are the differences in the new dataset (with generated data) from the original dataset?

Answer:

I use skit-learn train_test_split to get 25% data from the training set as a validation set.

data augmentation:

For every image randomly generate 10 augmented images

  • Motivation: During training, I found using simple LeNet I can easily overfit the original data set, therefore, data augmentation is helpful to get more data and hopefully get better results.

Note that, I think it's OK for different class have different number of training examples. The assumption is that the distribution of different will reflect the true distribution of all the traffic signs, therefore a bias towards larger number of specific class is actually good thing. Another concern of making number of classes equal is that augmented image can't catch some variations of true data, so for some example if you augment 10 times and have the same number of training examples as the larger class, you can't actually know whether the bias is the same. Therefore when doing data augmentation, I will uniformly augment every image.

Question 3

What does your final architecture look like? (Type of model, layers, sizes, connectivity, etc.) For reference on how to build a deep neural network using TensorFlow, see Deep Neural Network in TensorFlow from the classroom.

Answer:

In this project I tried two architecture, first is Lenet:

Architecture 1 LeNet

Layer 1: Convolutional. The output shape should be 28x28x6.

Activation. Relu

Pooling. The output shape should be 14x14x6.

Layer 2: Convolutional. The output shape should be 10x10x16.

Activation. Relu

Pooling. The output shape should be 5x5x16.

Flatten. Flatten the output shape of the final pooling layer such that it's 1D instead of 3D.

Layer 3: Fully Connected. This should have 120 outputs.

Activation. Relu

Layer 4: Fully Connected. This should have 84 outputs.

Activation. Relu

Layer 5: Fully Connected (Logits). This should have 10 outputs.

Output

Return the result of the 2nd fully connected layer.

The second one is similar to the paper http://people.idsia.ch/~juergen/ijcnn2011.pdf I add one more fully connected layer at the end.

Architecture 2

Layer 1: Convolutional. The output shape should be 28x28x100.

Activation. Relu

Pooling. The output shape should be 14x14x100.

Layer 2: Convolutional. The output shape should be 12x12x150.

Activation. Relu

Pooling. The output shape should be 6x6x150.

Layer 3: Convolutional. The output shape should be 5x5x250.

Activation. Relu

Pooling. The output shape should be 2x2x250.

Flatten. Flatten the output shape of the final pooling layer such that it's 1D instead of 3D.

Layer 4: Fully Connected. This should have 200 outputs.

Activation. Relu and Drop out

Layer 5: Fully Connected (Logits). This should have 43 outputs.

Output

Return the result of the 3nd fully connected layer.

Question 4

How did you train your model? (Type of optimizer, batch size, epochs, hyperparameters, etc.)

Answer:

  • Optimizer I choose Adam optimizer, it works well.
  • Batch size I tried batch size from 8 16, 32 ... to 1024 The smaller the batch size, the faster the network converges, however if the number is too small (like 8) It oscillating a lot and couldn't converge to a good result.
  • Epochs Since I am using my local machine, I want make the most use out of it, generally I set number of epochs to 50, also I implement my own version of early stop(not exactly the same as other's) basically monitoring the recent few training loss, if it is larger than the average of last 5 then stop.
  • Learning rate Learning rate is by far the most tricky one. I did randomize search between 10^-3 to 10^-7 for the larger network. I found something really strange for the large network. When my learning rate is > 10^-4. It almost always died at 0.057 accuracy. when I set between 10^-4-10^-5 sometimes dies sometimes doesn't and give a really good result within 50 EPOCH. (99.7% validation accuracy) Smaller 10^5 to 10^6 it almost always shows potential but too slow, only get 10% accuracy at 20% accuracy at 50 EPOCH. Also, with same initialization, I can get totally different behavior. I suppose this is something to do with the initialization, Maybe I'll try the Xavier if I have time later. I kept one of the experiment in this note book, please leave a comment if you know what's happening.
  • Dropout rate This is really powerful way to regularization, With even 0.99 rate the distance between training accuracy and testing accuracy decrease dramatically. Without it, if I let the network to train too long, it almost always end up overfitting. On the other hand if the dropout rate is too low, the system is very unstable. I choose 0.8 it performs well.

Question 5

What approach did you take in coming up with a solution to this problem? It may have been a process of trial and error, in which case, outline the steps you took to get to the final solution and why you chose those steps. Perhaps your solution involved an already well known implementation or architecture. In this case, discuss why you think this is suitable for the current problem.

Answer:

I tried LeNet first, it give me pretty good result with a short time, so I read more about the traffic sign classification, I tried various way of preprocessing, and built a larger network with more layers compare to LeNet, the idea is the task is basically similar to the task of identifying the handwritings. The main difference are:

  1. This project has more inputs and classes
  2. This project has color image instead of a grayscale image
  3. The images from this data set was taken in various lighting conditions, result in a data set has high variance in the contrast. After data augmentation, I found it's very likely to overfit data, so I add dropout.

Step 3: Test a Model on New Images

Take several pictures of traffic signs that you find on the web or around you (at least five), and run them through your classifier on your computer to produce example results. The classifier might not recognize some local signs but it could prove interesting nonetheless.

You may find signnames.csv useful as it contains mappings from the class id (integer) to the actual sign name.

Implementation

Use the code cell (or multiple code cells, if necessary) to implement the first step of your project. Once you have completed your implementation and are satisfied with the results, be sure to thoroughly answer the questions that follow.

In [ ]:
### Load the images and plot them here.
### Feel free to use as many code cells as needed.

Question 6

Choose five candidate images of traffic signs and provide them in the report. Are there any particular qualities of the image(s) that might make classification difficult? It could be helpful to plot the images in the notebook.

Answer:

In [99]:
# load test images
from skimage import io
import numpy as np
import matplotlib.image as mpimg


new_image = np.empty(shape=( 32, 32, 3), dtype=np.uint8)
raw_images=[]
transfered_images = []
l = [1,20,25,31,35]
test_imgs = []
for i in range(len(l)):
    new_image = cv2.resize(cv2.imread('/home/luke/Pictures/{}.png'.format(str(l[i]))),(32,32))
    raw_images.append(cv2.cvtColor(new_image, cv2.COLOR_BGR2RGB))
# src.create(32, 32, CV_8UC1);
# src = imread(cv2.resize(
#                     mpimg.imread('/home/luke/Pictures/{}.png'.format(str(l[i]))),(32,32)), CV_8UC1);
print(new_image.shape)
plt.figure(figsize=(16, 16))
for i in range(len(raw_images)):
    plt.subplot(8, 8, i+1)
    plt.axis('off')
    plt.imshow(raw_images[i])
    transfered_images.append(CLAHE_transform(raw_images[i]))
plt.figure(figsize=(16, 16))

for i in range(len(raw_images)):
    plt.subplot(8, 8, i+1)
    plt.axis('off')
    plt.imshow(transfered_images[i])    
(32, 32, 3)
In [73]:
prediction = tf.argmax(logits, 1)
In [105]:
with tf.Session() as sess:
    saver.restore(sess, tf.train.latest_checkpoint('.'))

    predict = sess.run(prediction, feed_dict={x: raw_images, keep_prob:1.0})
    predict1 = sess.run(prediction, feed_dict={x: transfered_images, keep_prob:1.0})

    print("Prediction with raw image:             ", predict)
    print("Prediction after CLAHE transformation: ", predict1)
    print("Ground Truth:                          ", l)
Prediction with raw image:              [ 1 30 25 31 35]
Prediction after CLAHE transformation:  [ 1 20 25 31 35]
Ground Truth:                           [1, 20, 25, 31, 35]

Question 7

Is your model able to perform equally well on captured pictures when compared to testing on the dataset? The simplest way to do this check the accuracy of the predictions. For example, if the model predicted 1 out of 5 signs correctly, it's 20% accurate.

NOTE: You could check the accuracy manually by using signnames.csv (same directory). This file has a mapping from the class id (0-42) to the corresponding sign name. So, you could take the class id the model outputs, lookup the name in signnames.csv and see if it matches the sign from the image.

Answer:

The model give a very good estimation on the real data, although the real data is kind different(blurred by resizing) It received 100% accuracy on 5 images.

However interestingly, I found that without doing transform, it doesn't give a 100% accuracy. But to my eyes, there is not much difference before or after transformation. Actually, it's easier for me to recognize the image without the transformation.

In [110]:
### Visualize the softmax probabilities here.
### Feel free to use as many code cells as needed.
def visulize_prediction(transfered_images):
    softmax_logits = tf.nn.softmax(logits)
    top_k = tf.nn.top_k(softmax_logits, k=3)
    with tf.Session() as sess:
        sess.run(tf.global_variables_initializer())
        saver = tf.train.import_meta_graph('./lenet.meta')
        saver.restore(sess, "./lenet")
        my_softmax_logits = sess.run(softmax_logits, feed_dict={x: transfered_images, keep_prob: 1.0})
        my_top_k = sess.run(top_k, feed_dict={x: transfered_images, keep_prob: 1.0})


        fig, axs = plt.subplots(len(transfered_images),4, figsize=(12, 14))
        fig.subplots_adjust(hspace = .4, wspace=.2)
        axs = axs.ravel()

        for i, image in enumerate(transfered_images):
            axs[4*i].axis('off')
            axs[4*i].imshow(image)
            axs[4*i].set_title('input')
            guess1 = my_top_k[1][i][0]
            index1 = np.argwhere(y_validation == guess1)[0]
            axs[4*i+1].axis('off')
            axs[4*i+1].imshow(X_validation[index1].squeeze())
            axs[4*i+1].set_title('top guess: {} ({:.2f}%)'.format(guess1, 100*my_top_k[0][i][0]))
            guess2 = my_top_k[1][i][1]
            index2 = np.argwhere(y_validation == guess2)[0]
            axs[4*i+2].axis('off')
            axs[4*i+2].imshow(X_validation[index2].squeeze())
            axs[4*i+2].set_title('2nd guess: {} ({:.2f}%)'.format(guess2, 100*my_top_k[0][i][1]))
            guess3 = my_top_k[1][i][2]
            index3 = np.argwhere(y_validation == guess3)[0]
            axs[4*i+3].axis('off')
            axs[4*i+3].imshow(X_validation[index3].squeeze())
            axs[4*i+3].set_title('3rd guess: {} ({:.2f}%)'.format(guess3, 100*my_top_k[0][i][2]))
visulize_prediction(transfered_images)
In [111]:
visulize_prediction(raw_images)

Question 8

Use the model's softmax probabilities to visualize the certainty of its predictions, tf.nn.top_k could prove helpful here. Which predictions is the model certain of? Uncertain? If the model was incorrect in its initial prediction, does the correct prediction appear in the top k? (k should be 5 at most)

tf.nn.top_k will return the values and indices (class ids) of the top k predictions. So if k=3, for each sign, it'll return the 3 largest probabilities (out of a possible 43) and the correspoding class ids.

Take this numpy array as an example:

# (5, 6) array
a = np.array([[ 0.24879643,  0.07032244,  0.12641572,  0.34763842,  0.07893497,
         0.12789202],
       [ 0.28086119,  0.27569815,  0.08594638,  0.0178669 ,  0.18063401,
         0.15899337],
       [ 0.26076848,  0.23664738,  0.08020603,  0.07001922,  0.1134371 ,
         0.23892179],
       [ 0.11943333,  0.29198961,  0.02605103,  0.26234032,  0.1351348 ,
         0.16505091],
       [ 0.09561176,  0.34396535,  0.0643941 ,  0.16240774,  0.24206137,
         0.09155967]])

Running it through sess.run(tf.nn.top_k(tf.constant(a), k=3)) produces:

TopKV2(values=array([[ 0.34763842,  0.24879643,  0.12789202],
       [ 0.28086119,  0.27569815,  0.18063401],
       [ 0.26076848,  0.23892179,  0.23664738],
       [ 0.29198961,  0.26234032,  0.16505091],
       [ 0.34396535,  0.24206137,  0.16240774]]), indices=array([[3, 0, 5],
       [0, 1, 4],
       [0, 5, 1],
       [1, 3, 5],
       [1, 4, 3]], dtype=int32))

Looking just at the first row we get [ 0.34763842, 0.24879643, 0.12789202], you can confirm these are the 3 largest probabilities in a. You'll also notice [3, 0, 5] are the corresponding indices.

Answer:

As the graph above shows, the network predict almost 100% accuracy for all the transformed images. That's reasonable, since these image are found in google search" Germen traffic signs" and the network is build based on the transfered image.

The interesting part is for the raw image. We can find most of the predictions are still accurate. However for the second one, although the true label is the second choice but it only get 2% probability. The reason I think is the watermark on the image, without the transform, the network confuse the watermark as some structure. After the transform however, the network seems to recognize it as noise.

Note: Once you have completed all of the code implementations and successfully answered each question above, you may finalize your work by exporting the iPython Notebook as an HTML document. You can do this by using the menu above and navigating to \n", "File -> Download as -> HTML (.html). Include the finished document along with this notebook as your submission.